{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM & Word Embeddings for Sentiment Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNNs are commonly applied to various natural language processing tasks. We've already encountered sentiment analysis using text data in part three of this book.\n",
    "\n",
    "We are now going to illustrate how to apply an RNN model to text data to detect positive or negative sentiment (which can easily be extended to a finer-grained sentiment scale). We are going to use word embeddings to represent the tokens in the documents. We covered word embeddings in Chapter 15, Word Embeddings. They are an excellent technique to convert text into a continuous vector representation such that the relative location of words in the latent space encodes useful semantic aspects based on the words' usage in context.\n",
    "\n",
    "We saw in the previous RNN example that Keras has a built-in embedding layer that allows us to train vector representations specific to the task at hand. Alternatively, we can use pretrained vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports & Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:17.846968Z",
     "start_time": "2020-06-22T00:56:16.226960Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from tensorflow.keras.datasets import imdb\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, GRU, Embedding\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "import tensorflow.keras.backend as K\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:17.880723Z",
     "start_time": "2020-06-22T00:56:17.848466Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using GPU\n"
     ]
    }
   ],
   "source": [
    "gpu_devices = tf.config.experimental.list_physical_devices('GPU')\n",
    "if gpu_devices:\n",
    "    print('Using GPU')\n",
    "    tf.config.experimental.set_memory_growth(gpu_devices[0], True)\n",
    "else:\n",
    "    print('Using CPU')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:17.897259Z",
     "start_time": "2020-06-22T00:56:17.883059Z"
    }
   },
   "outputs": [],
   "source": [
    "sns.set_style('whitegrid')\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:17.907985Z",
     "start_time": "2020-06-22T00:56:17.898465Z"
    }
   },
   "outputs": [],
   "source": [
    "results_path = Path('results', 'sentiment_imdb')\n",
    "if not results_path.exists():\n",
    "    results_path.mkdir(parents=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep the data manageable, we will illustrate this use case with the IMDB reviews dataset, which contains 50,000 positive and negative movie reviews evenly split into a train and a test set, and with balanced labels in each dataset. The vocabulary consists of 88,586 tokens.\n",
    "\n",
    "The dataset is bundled into Keras and can be loaded so that each review is represented as an integer-encoded sequence. We can limit the vocabulary to num_words while filtering out frequent and likely less informative words using skip_top, as well as sentences longer than maxlen. We can also choose oov_char, which represents tokens we chose to exclude from the vocabulary on frequency grounds, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:17.922921Z",
     "start_time": "2020-06-22T00:56:17.910357Z"
    }
   },
   "outputs": [],
   "source": [
    "vocab_size = 20000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:21.726976Z",
     "start_time": "2020-06-22T00:56:17.925579Z"
    }
   },
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(seed=42, \n",
    "                                                      skip_top=0,\n",
    "                                                      maxlen=None, \n",
    "                                                      oov_char=2, \n",
    "                                                      index_from=3,\n",
    "                                                      num_words=vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:22.627584Z",
     "start_time": "2020-06-22T00:56:21.728546Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD+CAYAAADWKtWTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xT530/8M/R1bIk36+AbYLBwUCCL7k30MadR5st2TIvsZOMrambLnttSfsL/aUsGYQSbmnT/NrlxbKFjnV12uIky9rSS5KBu7gh5GIHQQwCgjG+gO+2bEuyruf8/jAWGCzLNpaPpPN5v155Yfnokb7nRP748XOe8xxBkiQJRESkCCq5CyAiovnD0CciUhCGPhGRgjD0iYgUhKFPRKQgGrkLmIrFYoFer5e7jJjndrt5nGXE4y+fWD32brcbRUVFk26L6NDX6/UoLCyUu4yYZ7VaeZxlxOMvn1g99larNeg2Du8QESkIQ5+ISEEY+kRECsLQJyJSEIY+EZGCMPSJiBSEoU9EpCAMfSIiBWHoExEpSERfkUukRENOD0bcvlm1Nes1SIzXzXFFFEsY+kQRZsTtQ/3pvlm1XVuQxtCnKXF4h4hIQRj6REQKwtAnIlIQhj4RkYIw9ImilKXdhk/PD8ldBkUZzt4hikI9Iy78V2MH0sw63LAwUe5yKIqwp08UZSRJwi8tF+CXJPTbPRAlSe6SKIow9ImijKXdhpY+B3JT4uETJdicXrlLoijC0CeKIqMeP377aSdykg3445WZAIDeEbfMVVE0YegTRZG3j3fB6fHjz4oWIsMcBwDotTP0afp4IpcoSrQNOPHxuQHckZ+KBUkGSJIEg1aNPvb0aQbY0yeKAn5Rwi8t52GO0+CPCseGdQRBQLpZz54+zQhDnygKfHC2H51DLvzpjQug16oD308z6dnTpxlh6BNFuKFRL/7H2o2CTBNWLkiYsC3dpMOI2weX1y9TdRRtGPpEEe43xy5AFCXcu3ohBEGYsC3drAfAGTw0fQx9ogjmdPvQdGEYn1uahhTj1evkp5nGQr+P4/o0TQx9ogg25Bq78GpBkmHS7SkmHVQCp23S9DH0iSLY8OhY6CfGTT67WqNSITlex+EdmjaGPlEEG3aN3SvXbNAGfU66Wc/hHZo2hj5RBBu+OLxjDtLTB4B0k54Lr9G0MfSJItjwqA9GnRoaVfAf1TSznguv0bSFXIZBFEVs2bIFp06dgk6nw7Zt25CXlxfYXldXh927d0Oj0aCiogIPPPBA0DbHjx/HY489hsWLFwMAHnzwQdx9991h2zmiaDfi8iJhiqEdYKynD3DaJk1PyNA/cOAAPB4PamtrYbFYsGvXLrz88ssAAK/Xi507d+KNN96AwWDAgw8+iLvuugtHjhyZtM2JEyfwyCOP4Ktf/WrYd4woFgyPepEQN3Xop43P1ee4Pk1DyNBvbGzEmjVrAABFRUVoamoKbGtubkZubi4SE8fu3FNaWoqGhgZYLJZJ2zQ1NaGlpQUHDx5EXl4enn76aZhMpqDv7Xa7YbVaZ793NC0ul4vHWUZXHn+PxoTOrk4AgM3pRpJeCjyejCRJ0KsFtPUMor+vHyNdrWGvOVYo8bMfMvTtdvuEYFar1fD5fNBoNLDb7TCbzYFtRqMRdrs9aJsbb7wR999/P1atWoWXX34Zu3fvxre//e2g763X61FYWDjbfaNpslqtPM4yuvL4dww6kZ0lwS9KGPX2ISslEdlZmVO+RmbiKBx+AalpqViUnBPukmNGrH72p/pFFvJErslkgsPhCDwWRREajWbSbQ6HA2azOWib8vJyrFq1CgBQXl6OEydOzHxviBRixOWFBIQc3gG48BpNX8jQLykpQX19PQDAYrGgoKAgsC0/Px+tra2w2WzweDxoaGhAcXFx0DbV1dU4duwYAODw4cNYuXLlnO8QUawYuThHP2GK6Zrj0s16jLh9sLt94S6LolzIT1N5eTkOHTqEqqoqSJKEHTt2YP/+/XA6naisrMTGjRtRXV0NSZJQUVGBzMzMSdsAwJYtW/Dcc89Bq9UiLS0Nzz33XNh3kChaDV28GjfU7B1gbLVNAGjrd2J5VkKIZ5OShQx9lUqFrVu3Tvhefn5+4OuysjKUlZWFbAMAK1euxL59+2ZbK5GijEzjwqxx4wuvtQ04w1oTRT9enEUUoYZdPqgEwKgPHfrjC68x9CkUhj5RhBoe9cIcp4XqijX0JzO+8FpbP0OfpsbQJ4pQIy7ftE7ijks369nTp5AY+kQRamgaSzBcLt2kR8fgKPwiF16j4Bj6RBFqxDU2vDNd6WY9PH4R5wdHw1gVRTuGPlEE8vhEuLxi0JunTGZ8Bk9znz1cZVEMYOgTRaDAOvozGN4ZX3ituYehT8Ex9Iki0PhtEqezBMM4o06NhDgNzvY5Qj+ZFIuhTxSBhmewBMM4QRCQmxKPs73s6VNwDH2iCDR+Ne5MZu8AQG5qPJp72dOn4Bj6RBFoeNQLnVoFvWZmP6K5KfHoHXEHfmkQXYmhTxSBhl0+mOM0EKZxNe7lclPiAQBn2dunIBj6RBFoeHRmF2aNW5RsAACc62fo0+QY+kQRaNjlndFJ3HELksZCv53LMVAQDH2iCCNJ0ti6O7Po6cdp1cgw69HKhdcoCIY+UYQZdvngE6UZzdG/XF5qPFrZ06cgGPpEEabPPnav2+ncPGUyOSnxXGKZgmLoE0WY3os3OE+cxfAOAOSlGNE17ILL65/LsihGMPSJIkyf3QMAM1ph83J5qWPTNjsG2dunqzH0iSJM38We/mxm7wBjwzsAeDKXJsXQJ4owfXY34nVqaNSz+/Ec7+nzLlo0GYY+UYTps3tmPXMHAFKNOhh1avb0aVIMfaII02t3I8Ewu6Ed4OJqm6lG9vRpUgx9ogjTZ3fP+iTuuNwUA1q5FANNgqFPFEF8fhGDjmsb3gGAvFQj2gdHIfIm6XQFhj5RBOmzeyBKuKbhHWBstU2PT0T3iGuOKqNYwdAniiDdw2Mhfe09fU7bpMkx9IkiSNcchf74uvpcjoGuFDL0RVHE5s2bUVlZifXr16O1tXXC9rq6OlRUVKCyshKvvfbatNrs378flZWVc7gbRLGhZzz0r3F4Z0GSAWqVwBk8dJWQoX/gwAF4PB7U1tZiw4YN2LVrV2Cb1+vFzp07sXfvXtTU1KC2tha9vb1TtrFarXjjjTcgSTzBRHSlrmEX1IIAo/7aQl+rVmFhkoGrbdJVQn6yGhsbsWbNGgBAUVERmpqaAtuam5uRm5uLxMREAEBpaSkaGhpgsVgmbTM4OIgXXngBTz/9NDZt2hSyOLfbDavVOvO9ohlxuVw8zjK6/PifbutBkkGD7u6uWb1Wf4qAka6xv6xT9RJOdfTz/+0UlPjZDxn6drsdJpMp8FitVsPn80Gj0cBut8NsNge2GY1G2O32Sdt4PB4888wzePrpp6HX66dVnF6vR2Fh4Uz2h2bBarXyOMvo8uPven8YGYkCsrOyZ/VaqWmpWJScAwAoPOnD7z7t5P/bKcTqZ3+qX2Qhh3dMJhMcjksXeYiiCI1GM+k2h8MBs9k8aZuTJ0+itbUVW7ZswZNPPokzZ85g+/bts9oholjVPexCmml6naJQ8lLiMej0YtjlnZPXo9gQMvRLSkpQX18PALBYLCgoKAhsy8/PR2trK2w2GzweDxoaGlBcXDxpmxtvvBG/+c1vUFNTgxdffBFLly7FM888E6bdIopO3cNupJvnKPRTOYOHrhZyeKe8vByHDh1CVVUVJEnCjh07sH//fjidTlRWVmLjxo2orq6GJEmoqKhAZmbmpG2IaGourx9Do16kmXRz8nqXL7G8amHinLwmRb+Qoa9SqbB169YJ38vPzw98XVZWhrKyspBtLrdo0aLA9E4iGjN+YVb6XA3vpBoBcIllmogXZxFFiK6hsdCfqzF9k16DVKMObQNceI0uYegTRYjui3fMSp2j0AfGhni4FANdjqFPFCHGr8ZNN8/NmD4wdjKXoU+XY+gTRYiuIRfitCqYrvFq3MvlpcSjc2gUHp84Z69J0Y2hTxQhukfcyEqIgyAIc/aaualGiBJw3jY6Z69J0Y2hTxQhuodcyEiIm9PXzA1M2+TJXBrD0CeKEN0jLmTOceiPX6DVzmmbdBFDnygCSJKEriEXshLmbuYOAGSY9dBrVDyZSwEMfaIIMDzqg9snznlPXxAE5KbEc4llCmDoE0WA8XvZznXoA2NDPFx/h8Yx9IkiwPjVuOEI/dwUI9oGnLxxEQFg6BNFhPF1d7LCEPqL0+Ix6vWje9g9569N0WfurgIholkbD/2MBD367LMPZ59fRMfgxKGcRMPYTdYPn+3DzYtTJmwz6zVIjJ+7K4Ap8jH0iSJA97AbiQYt4rTqa3qdUa+II80DE743cvEmKgetPRj1TLwyd21BGkNfYTi8QxQBuoddYRnaAcZW2zRo1egZ4fAOMfSJIkL3sAsZczxHf5wgCMgw69HDMX0CQ58oInQPu8PW0weAdLMePRenhZKyMfSJZOYXJfTa3WGZrjkuIyEOTo8fdrcvbO9B0YGhTySzIZcfflFCZmIYQ//izdbZ2yeGPpHM+px+AECmOTxj+sCl0O/lyVzFY+gTyWzAOTbkEs7hnUSDFjqNiidziaFPJLf+0bGeflYYh3cCM3g4vKN4DH0imXXbvdCoBKQaw3uR1Fjos6evdAx9Ipmd7nNjebYZGnV4fxwzzHEYcfkw6vGH9X0osjH0iWQkihJO97tRlJMU9vfiDB4CGPpEsjrbZ8eoV8LqRfMQ+hdPFHMGj7Ix9IlkdKTNBgAozg1/6CfFa6FVCxzXVziGPpGMjnbYYNAKWJJmCvt7qQQB6SbO4FG6kEsri6KILVu24NSpU9DpdNi2bRvy8vIC2+vq6rB7925oNBpUVFTggQceCNrmzJkz2LRpEyRJwvLly7Fp0yao1de2lCxRNLO021CQqodKJczL+6Wb9bxJusKF7OkfOHAAHo8HtbW12LBhA3bt2hXY5vV6sXPnTuzduxc1NTWora1Fb29v0DYvvvginnzySezbtw8ulwt1dXXh2zOiCOfy+nGycwTL08M3P/9KGQlxsI164fZyBo9ShezpNzY2Ys2aNQCAoqIiNDU1BbY1NzcjNzcXiYmJAIDS0lI0NDTAYrFM2uall16CWq2Gx+NBb28vUlNT53yHiKLF8QtD8IkSCtLCt/zClQLLMdjdWJQcP2/vS5EjZOjb7XaYTJfGG9VqNXw+HzQaDex2O8xmc2Cb0WiE3W6fss358+fxyCOPwGQy4brrrpvyvd1uN6xW62z2i2bA5XLxOMvg7RNDAIDFZkw4/h6NCZ1dnbN6zeVpuinbChdX2Tzd3g21Ow79KQJGulpn9V6xQImf/ZChbzKZ4HA4Ao9FUYRGo5l0m8PhgNlsnrLNwoUL8c477+D111/Hrl278Pzzzwd9b71ej8LCwpnvFc2I1WrlcZbBv1iOIDsxDguSjROOf8egE9lZ0qxe0xAfj+ys7KDbM0QJ6qND8KgMyM7KQmpaKhYl58zqvWJBrH72p/pFFnJMv6SkBPX19QAAi8WCgoKCwLb8/Hy0trbCZrPB4/GgoaEBxcXFQds89thjOHfuHICxvwpUKk4eIuU62m6bl4uyLqdWCUg16TiDR8FC9vTLy8tx6NAhVFVVQZIk7NixA/v374fT6URlZSU2btyI6upqSJKEiooKZGZmTtoGAL7+9a9j48aN0Gq1MBgM2LZtW9h3kCgSDTg8aBtw4qFbcwF45vW9MxLicME2Oq/vSZEjZOirVCps3bp1wvfy8/MDX5eVlaGsrCxkG2Dsr4Z9+/bNtlaimHG0feyirNWLkgBPz7y+d4ZZj+Pnh+D1i/P6vhQZOL5CJANLuw0qAbhxUeK8v3eGWQ8JXI5BqRj6RDKwtNuwLMMMoz7kH9tzbnwNHo7rKxNDn2ieSZKEox3zfxJ3XLpJD41KwAUbQ1+JGPpE86y13wmb04vVMoW+WiVgQZIB7YNcjkGJGPpE8+xox9hJXLl6+gCwKNmAC7ZR+ESezFUahj7RPLO022DQqlGQGf6VNYNZlGyA1y/hXB97+0rD0CeaZ5Z2G25YmBj22yNOZXzdHWvnsGw1kDwY+kTzyOMTcfzCMFbnzP9UzculGnWI06pwgqGvOAx9onl0smsYHp8o20nccYIgICc5HtbOEVnroPnH0CeaR+NX4sp5EnfcomQDWnodGPVwbX0lYegTzaMj7TakmXRYmGSQuxQsSo6HX5Jw/MKQ3KXQPGLoE82j8ZU1BWF+bo84lUXJY794LBf/+iBlYOgTzZOhUS+aex1ji6xFAHOcFhlmPY52sKevJAx9onny6cVwLcqNjNAHgBXZCYHzDKQMDH2ieTJ+Je6NEdLTB4DCBWa0DTgx6JjfNf1JPgx9onlypM2GJelGJBq0cpcSUJidAODSLySKfQx9onkgSRIs7TYURVAvHwCWZ5ohCMDRdo7rKwVDn2geXBhyoc/ulv2irCvF6zVYmm5iT19BGPpE8yCSLsq60uqcJBzrsEGSJLlLoXnA0CeaB5Z2G3RqFZZnm+Uu5Sqrc5LQZ/fgPG+WrggMfaJ5YGm3YcWCBOg1arlLuUrxxb8+Gs4NylwJzQeGPlGY+fwiPu0YisihHWBsrn6aSYeDJ3vkLoXmAUOfKMw+67Fj1OuP2NBXqQTcdX0G3j3VA6+fd9KKdQx9ojAbP4kbaTN3LvfFwkwMu3wc4lEAhj5RmFnabUg0aLE4NV7uUoJasywNOrUKB63dcpdCYcbQJwozS7sNqyNkZc1gjHoNbs9P5bi+AjD0icLI4fbhdPcIihbJe3vEYHx+ER2DTnQMOlGSl4SWPgfeP9MX+N6Qk2vyxBqN3AUQxbKm80MQpchaWfNyo14RR5oHAAAa1VgfsOaDVqxZlg4AWFuQhsR4nWz10dwL2dMXRRGbN29GZWUl1q9fj9bW1gnb6+rqUFFRgcrKSrz22mtTtrFarXjooYewfv16VFdXo6+vLwy7RBQ5joyfxI2wNXcmkxyvQ1ZCHO+bG+NChv6BAwfg8XhQW1uLDRs2YNeuXYFtXq8XO3fuxN69e1FTU4Pa2lr09vYGbbN9+3Zs2rQJNTU1KC8vx549e8K3Z0QyO9fnwL+924yVCxKQatLLXc60LM82o23AAafHJ3cpFCYhQ7+xsRFr1qwBABQVFaGpqSmwrbm5Gbm5uUhMTIROp0NpaSkaGhqCtnnxxRdRWFgIAPD7/dDro+MHgWimhpxefPU/P4YEYPdDJXKXM22FWQkQJeB0t13uUihMQo7p2+12mEymwGO1Wg2fzweNRgO73Q6z+dJaIkajEXa7PWibjIwMAMAnn3yCV199FT/96U+nfG+32w2r1TrjnaKZcblcPM7TpNIb4fJPPQvHJ0r4ztstaOt34jtfug5DgwM4OjgQ/PnQ4+hn7YHHglqLzq7OWdW3PE03o7ZXPl8tSTBoBRxp6UamdhT9KQJGulqneIXopsTPfsjQN5lMcDgcgceiKEKj0Uy6zeFwwGw2T9nmt7/9LV5++WW88sorSElJmfK99Xp94C8DCh+r1crjPE0dg040ng5+LkqSJPzCcgHHLthRUbIIgt6I4wNTr17Z2dWL7KzswOPi3PgJj2fCED+ztpM9f0W2iOOdQ8jIyEJqWioWJefMqpZoEKuf/al+kYUc3ikpKUF9fT0AwGKxoKCgILAtPz8fra2tsNls8Hg8aGhoQHFxcdA2v/zlL/Hqq6+ipqYGOTmx+0Ei5TrU3I+Pzw3g8wXpKM1LlrucWVmebYbLK+JcvyP0kynqhOzpl5eX49ChQ6iqqoIkSdixYwf2798Pp9OJyspKbNy4EdXV1ZAkCRUVFcjMzJy0jd/vx/bt25GdnY3HH38cAHDzzTfjiSeeCPtOEs0Ha+cwfvdpJ1YuSED5iky5y5m1pRkmqFUCTnYOy10KhUHI0FepVNi6deuE7+Xn5we+LisrQ1lZWcg2APDRRx/Ntk6iiNY5NIraj9uxIMmA+0tzoIrgq29D0WvUyE834mTXCG+sEoN4RS7RNRp2efGTw60w6NRYf1sedJro/7FanpWAfocH7QO8sUqsif5PJ5GMPD4Rr37QilGPH+tvy0OCQSt3SXNiedbYrLz3zvACyljD0CeaJVGS8EZjO84PjqLy5hwsSDLIXdKcSYrXITsxDu83M/RjDUOfaJYOWLvRdGEYX1qVhcLsBLnLmXPXZ5nxaccwbFx0LaYw9Ilm4ZO2QfzvqV7cvDgZdy5Nk7ucsCjMSoBfkvC/p3rlLoXmEEOfaIZa+hz470/OY0m6EfeuXhjR6+Rfi4XJBqQYdTjAG6vEFIY+0Qx0Dbnw0w9bkWzU4uFb8qBWxWbgA4BKEHD7klS8e7qX986NIQx9omnyixK2/voE/KKEv759MQw6tdwlhd3nlqZixOXDRy3B1w6i6MLQJ5qmf/n9GRzrGMK9qxcgLUqWSr5Wt1yXgjitCm8f75K7FJojDH2iaTjSNogfHPwM5SsyUZwbnWvqzEacVo21y9LxzvFuiCKvzo0FDH2iEOxuH75Za0FWQhyeLF8mdznzbt3KLHQNu3Ds/JDcpdAcYOgThfCdXx1H+4AT/6+yCOa42Ljidia+WJgBjUrgEE+MYOgTTeE3xzrxemMH/v6upbjluqnv/xCrkuJ1uG1JKt5u6uICbDGAoU8UxAXbKP7xzWMoyknCE19U3rDO5datzMTZPgfO9PA2itGOoU80Cb8o4f/UWuAXJfywqghatbJ/VMpXZAEAh3higLI/yURBvFJ/Fh+2DGDLvSuRl2qUuxzZZSXGoSgnCW8f59W50Y6hT3SFTzuG8P13TuFPbsjGX5YukruciLFuZRY+PT+E9gGn3KXQNWDoE13G6fHhG/uOIN2sx/b7VsXsujrT5fOL6Bh0omPQiZsXJ0OjEvC9t08FvjfEFTijTsjbJRIpyXO/PoGWfgd++rVbkRSvk7sc2Y16RRxpvrQEw82LU/DrYxewJN2IDHMc1hakIZHHKaqwp0900VtNXfj5R+3427X5uCM/NpdLvlZ3Lc+ARq3C/5zg2H60YugTAegedmHjm8dww8JEPFleIHc5Ecuk12DNsjQcvzCMNo7tRyWGPimeKErY8NpRuL0iflBVFBM3Ng+nO5emwaTX4K2mTl6sFYX46SbF+/f3WvDemT5svmcF8tNNcpcT8fQaNcqWZ+BcvxPvN/fLXQ7NEEOfFO34hSF89+2T+OMVmai6OUfucqLGzYtTkGrU4V/fPQs/V9+MKgx9UqxRjx/f2GdBcrwOuypuVPz0zJlQqwT88costPQ58OYnHXKXQzPA0CfF2vFbK8702PHiA0VIMXLa4UytWpCAwmwzXvyf03B5/XKXQ9PE0CdFOnCiGzUftOLRNdfhzmWcnjkbgiDg776Qj84hF378/jm5y6FpYuiT4vSMuPDUfx1DYXYCvrXuernLiWolucn4wvXp+Jffn4GNV+dGBYY+KYooSvjW68fgcPvwz1VF0Gti/+bm4bbxy8thd/uw+/dn5C6FpiFk6IuiiM2bN6OyshLr169Ha2vrhO11dXWoqKhAZWUlXnvttWm12bFjB37+85/P4W4QTc8PD36G+tO9+Kc/KcSyTLPc5UQ9n1+ESa/Bl1dl48fvn8NHLf1ckyfChQz9AwcOwOPxoLa2Fhs2bMCuXbsC27xeL3bu3Im9e/eipqYGtbW16O3tDdpmYGAAX/va11BXVxe+PSKahCRJ+N7bJ/HDg5/hL0oW4q9uy5O7pJgw6hVRf7oPqxYmAgC2/8aK+tN9GHH7ZK6Mggm54FpjYyPWrFkDACgqKkJTU1NgW3NzM3Jzc5GYOPY/vLS0FA0NDbBYLJO2cTgcePzxx1FfXz+t4txuN6xW68z2iGbM5XLF9HGWJAmvfNyPX1iH8aVlZlSv0uHkyZOzei2PxoTOrs45rc/n9U54zeVpulm/x0zbhnr+TLavzopDQ8cQCpLasDJFwEhXa9B2kSLWP/uTCRn6drsdJtOlqxTVajV8Ph80Gg3sdjvM5kt/IhuNRtjt9qBtcnJykJOTM+3Q1+v1KCwsnMn+0CxYrdaYPc6iKOGfftmEX1iH8ZU7FuPZe1Zc03z8jkEnsrPm9mKkzq5OZGdlBx4b4uMnPJ6JmbYN9fyZbL871Y/T/afxXrsbf/+lZCxOi/yL3WL1sz/VL7KQwzsmkwkOhyPwWBRFaDSaSbc5HA6YzeYp2xDNF59fxLdeP4qffdiGv/tC/jUHPk0tTqvGvasXoHPIhdqGdrnLoSBChn5JSUmgZ26xWFBQcGkFwvz8fLS2tsJms8Hj8aChoQHFxcVTtiGaDx6fiG/ss+DNI+exobwAT627noE/D1YsSMSK7ATsfe8c2vq5CmckCtn9Li8vx6FDh1BVVQVJkrBjxw7s378fTqcTlZWV2LhxI6qrqyFJEioqKpCZmTlpG6L54vL68Q8/+wQHrD145u5CPLp2idwlKco9qxfgpbrP8PxbJ7H74RK5y6ErhAx9lUqFrVu3Tvhefn5+4OuysjKUlZWFbHO5xx9/fKZ1Ek3LqMePr9c04A+f9eG5P1uJ9bcvlrskxUk0aHFf8UL8/KM2tA84kZMSL3dJdBlenEUxw+724W/2foRDZ/rwvb+8kYEvo4qShVAJAvYeapG7FLoCQ59iwpDTi4d/9CEa2wbxg6pi3H9T5M8ciWUZCXG4Z/UCvPZxO4ZGvXKXQ5fhlBq6ypDTE1UX1ww6PXiy9ijO9Tuw7c9XoiQ3CR2D4TmJ6OZqktPi84u4d3U2/vvIefzbu8146NZcmPUa3kQ9AjD06Sojbh/qT/fJXca0DLu82PteCwYcHvzVbXmQJCGstRfnJoXttWPJqFdE55Ab+elG/Pj9c0iI0+JPV2cz9CMAh3coatmcHuypPwub04uv3LEYBVxLJ+Lcc+MCeP0iXmto5x22IgRDn6JSv92NV/5wFg6PD1/93GIs4b1tI1JGQhzuXb0AZ/scqDkc+csyKAFDn6JOz4gLe/5wFh6fiOo7lyA31Sh3STSFktxkFOUk4d/fa8HvT/XIXY7iMfQpqnQOjWJP/VmIEv0F5skAAAqrSURBVPC1NUuwMMkgd0kUgiAI+POihViaYcITPz+C5l673CUpGkOfokbHoBM/+kMLNGoVvr5mCbIS4uQuiaZJp1Fh51/cAJ1ahUf/swF9drfcJSkWQ5+iwrk+B/79vRbEaVV4dM0SpJn1cpdEM5Rm0uG5P1+J87ZRVL3yAU5cGOLNVmTA0KeId6bHjv94vwXmOA2+vjYfKUZO+4tGo14RNqcPD92Si5Y+Bx79SSPO20blLktxGPoU0U51DeMnh88hxajDo2uWINGglbskukbLMs146JZcdA278MQ+C4d65hlDnyJW0/khvPpBGzIS9Hj0ziUwxzHwY0VhdgLW35aH9gEnHt7zIWwc5pk3DH2KSJb2Qez7uA0Lkw2o/twSxOt58XisKcg04/mKG9DS58BXf/wxnJ7oWfojmjH0KeJ8fG4Arzd0YHGqEY98bjEMOrXcJVGYFOUk4dl7VsDSbsPf1jRi0MGhnnBj6FNEeb+5D/995DyWZZrwN3cshl7DwI9lo14RgiDgy6uy8YfP+vDPB8/IXVLM49/MFDHePd2Lt493YUV2AqpuzoFGzT6JUtyRn4rOIRf+4/1zSIzX4htfXMbbW4YJQ59kJ0kSDp7sQd3JHty4KBH3l+ZAreIPvJKMXbW7AJkJevzgwGf4rMeOF/5yNYf2woChT2HnE0XYXT7Y3T7YXT6MuH0Ycflgd3sx4vLB5vTivG0UpXnJuK947I5LpDwatQpPrSvAdWlGvPy/zTjTY8dLVUUoyEqQu7SYwtCnWRElCU6P/2KIe8f+vRjsIy4vRsYD3uXDaJAbjxi0apjjNDDFafBHhZn4wvXpDHyFc/kkLEqOx1/dlofahnY8+KMP8VJVMW5dksq//uYIQ58CJEmCw+NH+4ATLX2OQICP984vD3i724fJlkfXqgWY47Qw6TVIN+txXZoR5jgNzHotTHGasZDXj/3HMXsKpjA7AY99Ph+vN7TjoR99iBSjDk+WF+DhW3M51n+NGPoK4Pb50Wf3oHfEPfE/uwu9I2609dhg39+JvhHPpL1ylYCxoL4Y3tmJBpjHH18M+LFg10CnUfGHkuZEVkIc9vx1KT5qGcCvj3Xin37RhN992onvP1CErEQutjdbDP0o5RclDDqvCHL7laE+9m+wG1Mnx2uRbtYjXqPC9QuTkW7WI92sh1ol4Pyg62LIa2DQqTnsQrJQq1SI02rwFyWLsDA5Hu8c78KXfliP//jKzSjOTZa7vKjE0I8gkiRhxO27qkfeZ7861PsdnklvP2fQqpGRoEe6SY9lGSbckZ+KdJM+EOjj/6Ua9dBpxoZXrFYrCgsLA6/RMeiMmnvkkjKoBAG3L0nFA6ULsfHNJjz0ow/x1Lrr8de3L+ZY/wwx9OeBy+u/FNxT9Mh7R9xw+8Sr2mtUQiCssxLicMPCxEsBbtIj7eK/6WY9jFyugGJYmjkO62/Lw88+asN39p/A82+dxOpFSbj/phzcfUMW4nX8/IfCIzRLflHCgMMzRYi7Ao+HXZOvKZJi1AXCevFiYyDEJ/TKTXokGrRQsTdDBABIMGjxt2uXAJBwtGMI7zf341uvH8V3fnUc9xQtQOVNOchNiefPTRAM/ctIkoRhl29aPfIBh3vS2Svjs1bSTXosz0rAnUt1V4R43NjwikkHLWevEM2KIAgozk2GIKiwelESTHo1fne8G//V2IGffdgGAEg0aPHlVVn42prrsDTDfNVrNLYOYP+ng9hwnVdRK7gqIvRdXn/IEB9/7JlkeEWrFgI98IVJcSjKSZykRx6HNLOOf14SzTNBEFCQlQCHR8Qti1Pg8fnh9PhxpseON4+cx76P23Hn0lTcf1PO2CyzOC3qTvbg395thigBH3d9gP3/cKdi/ioImVCiKGLLli04deoUdDodtm3bhry8vMD2uro67N69GxqNBhUVFXjggQeCtmltbcXGjRshCAKWLVuGZ599FipV8N5uc68dm999f1Y7JkoIzG4ZmWR4RRCAlPhLvfAlacarhlXGv040aDkNkSgKxGnVuD0/FUfabPjyDdkozk3Ghy39aGwdxHv7LBOee/cNWcg1q/Gv75/HTw6fw33FC5EYH/t3ZQsZ+gcOHIDH40FtbS0sFgt27dqFl19+GQDg9Xqxc+dOvPHGGzAYDHjwwQdx11134ciRI5O22blzJ775zW/i1ltvxebNm3Hw4EGUl5cHfW8BmPUQiCCMXeCxdtnVY+QZZj1SjDpeHEQU44x6DcqWZ+Kbf1SAt493oTDbDIfbj3SzDhdsbpzvvIDFqfH43junYdRrsDTDBK1aBa1aBY1agO7ivxqVCioBgc6fShhbNkKjEqASBEiQIF0x3CsIgAABoiTBL0pQX3zudPuPGpUQls5myNBvbGzEmjVrAABFRUVoamoKbGtubkZubi4SExMBAKWlpWhoaIDFYpm0zfHjx3HLLbcAANauXYtDhw5NGfpL0k342drCoNuJiKZDrRKQnWjAdWkmHGmzITPBgAs2N1SCgH+8uxDf3GfB/33jmNxlTvCVOxZjy70r5/x1Q4a+3W6HyWQKPFar1fD5fNBoNLDb7TCbL50gMRqNsNvtQdtIkhT4zWU0GjEyMjLle7vdblit1hnvFM3clce5mGtcTc7mmPNjU5yQAMAxN+8x07ahnj/b7ZN9fzrfm8njUF9P9b2L/44d+0G8+dClIetIMtv8c7uD34wmZOibTCY4HJc+kKIoQqPRTLrN4XDAbDYHbXP5+L3D4UBCwtSfzqKiolDlERHRDIQc1C4pKUF9fT0AwGKxoKCgILAtPz8fra2tsNls8Hg8aGhoQHFxcdA2K1aswIcffggAqK+vx0033TTnO0RERMEJknTl6YeJxmfinD59GpIkYceOHThx4gScTicqKysDs3ckSUJFRQUefvjhSdvk5+ejpaUFmzZtgtfrxZIlS7Bt2zao1bxJAhHRfAkZ+kREFDs4Z5GISEEY+kRECsLQJyJSEIY+EZGCMPRpUocPH8YzzzwjdxmKc/jwYTz11FN44okncPLkSbnLUZympiZs2LAB3/72t9HXF5s3EmLo01VaW1tx4sSJKa/qo/AYHR3F888/j8ceewzvvfee3OUojtvtxrPPPovPf/7zsFgsoRtEIYY+XSUvLw/V1dVyl6FIZWVlGB0dRU1NDe677z65y1Gc0tJSnDlzBnv37p1wC9FYwtAniiCDg4PYvn07nnjiCaSmpspdjuIcO3YMq1atwp49e/Dqq6/KXU5YMPQV5ujRo1i/fj2AsautN2/ejMrKSqxfvx6tra0yVxfbpnPsd+7cie7ubnz/+9/HW2+9JWe5MWc6x9/hcODpp5/Gtm3bsG7dOjnLDRve5klB9uzZg1/96lcwGAwApr5XAgC88MILcpUac6Z77L/73e/KXGlsmu7xv/3223H77bfLXG14saevILm5uXjppZcCj6e6VwLNLR57efH4X8LQV5B169YFlsUGgt8rgeYej728ePwvYegr2FT3SqDw4rGXl5KPP0Nfwaa6VwKFF4+9vJR8/JXxq40mVV5ejkOHDqGqqipw3wOaHzz28lLy8ed6+kRECsLhHSIiBWHoExEpCEOfiEhBGPpERArC0CciUhCGPhGRgjD0iYgUhKFPRKQgDH0iIgX5/0miK1WeaVLTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = sns.distplot([len(review) for review in X_train])\n",
    "ax.set(xscale='log');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the second step, convert the lists of integers into fixed-size arrays that we can stack and provide as input to our RNN. The pad_sequence function produces arrays of equal length, truncated, and padded to conform to maxlen, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:22.631479Z",
     "start_time": "2020-06-22T00:56:22.629298Z"
    }
   },
   "outputs": [],
   "source": [
    "maxlen = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:23.122747Z",
     "start_time": "2020-06-22T00:56:22.632757Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train_padded = pad_sequences(X_train, \n",
    "                        truncating='pre', \n",
    "                        padding='pre', \n",
    "                        maxlen=maxlen)\n",
    "\n",
    "X_test_padded = pad_sequences(X_test, \n",
    "                       truncating='pre', \n",
    "                       padding='pre', \n",
    "                       maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:23.127132Z",
     "start_time": "2020-06-22T00:56:23.123708Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((25000, 100), (25000, 100))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_padded.shape, X_test_padded.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Model Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define our RNN architecture. The first layer learns the word embeddings. We define the embedding dimension as previously using the input_dim keyword to set the number of tokens that we need to embed, the output_dim keyword, which defines the size of each embedding, and how long each input sequence is going to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:23.138436Z",
     "start_time": "2020-06-22T00:56:23.129179Z"
    }
   },
   "outputs": [],
   "source": [
    "K.clear_session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Loss Metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:23.145022Z",
     "start_time": "2020-06-22T00:56:23.139827Z"
    }
   },
   "outputs": [],
   "source": [
    "embedding_size = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we are using GRUs this time, which train faster and perform better on smaller data. We are also using dropout for regularization, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:24.321967Z",
     "start_time": "2020-06-22T00:56:23.146967Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer gru will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 100, 100)          2000000   \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (None, 32)                12864     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 2,012,897\n",
      "Trainable params: 2,012,897\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "rnn = Sequential([\n",
    "    Embedding(input_dim=vocab_size, \n",
    "              output_dim= embedding_size, \n",
    "              input_length=maxlen),\n",
    "    GRU(units=32,  \n",
    "        dropout=0.2, # comment out to use optimized GPU implementation\n",
    "        recurrent_dropout=0.2),\n",
    "    Dense(1, activation='sigmoid')\n",
    "])\n",
    "rnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting model has over 2 million parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compile the model to use our custom AUC metric, which we introduced previously, and train with early stopping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:24.357020Z",
     "start_time": "2020-06-22T00:56:24.323139Z"
    }
   },
   "outputs": [],
   "source": [
    "rnn.compile(loss='binary_crossentropy', \n",
    "            optimizer='RMSProp', \n",
    "            metrics=['accuracy', \n",
    "                     tf.keras.metrics.AUC(name='AUC')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:24.361309Z",
     "start_time": "2020-06-22T00:56:24.358612Z"
    }
   },
   "outputs": [],
   "source": [
    "rnn_path = (results_path / 'lstm.h5').as_posix()\n",
    "\n",
    "checkpointer = ModelCheckpoint(filepath=rnn_path,\n",
    "                               verbose=1,\n",
    "                               monitor='val_AUC',\n",
    "                               mode='max',\n",
    "                               save_best_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T00:56:24.369867Z",
     "start_time": "2020-06-22T00:56:24.362390Z"
    }
   },
   "outputs": [],
   "source": [
    "early_stopping = EarlyStopping(monitor='val_AUC', \n",
    "                               mode='max',\n",
    "                              patience=5,\n",
    "                              restore_best_weights=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training stops after eight epochs and we recover the weights for the best models to find a high test AUC of 0.9346:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T01:37:03.531798Z",
     "start_time": "2020-06-22T00:56:24.373220Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.4405 - accuracy: 0.7858 - AUC: 0.8756\n",
      "Epoch 00001: val_AUC improved from -inf to 0.92333, saving model to results/sentiment_imdb/lstm.h5\n",
      "782/782 [==============================] - 423s 541ms/step - loss: 0.4405 - accuracy: 0.7858 - AUC: 0.8756 - val_loss: 0.3949 - val_accuracy: 0.8205 - val_AUC: 0.9233\n",
      "Epoch 2/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.2879 - accuracy: 0.8835 - AUC: 0.9492\n",
      "Epoch 00002: val_AUC improved from 0.92333 to 0.93434, saving model to results/sentiment_imdb/lstm.h5\n",
      "782/782 [==============================] - 361s 461ms/step - loss: 0.2879 - accuracy: 0.8835 - AUC: 0.9492 - val_loss: 0.4025 - val_accuracy: 0.8404 - val_AUC: 0.9343\n",
      "Epoch 3/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.2407 - accuracy: 0.9039 - AUC: 0.9644\n",
      "Epoch 00003: val_AUC improved from 0.93434 to 0.93923, saving model to results/sentiment_imdb/lstm.h5\n",
      "782/782 [==============================] - 353s 451ms/step - loss: 0.2407 - accuracy: 0.9039 - AUC: 0.9644 - val_loss: 0.3568 - val_accuracy: 0.8499 - val_AUC: 0.9392\n",
      "Epoch 4/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.2098 - accuracy: 0.9177 - AUC: 0.9728\n",
      "Epoch 00004: val_AUC improved from 0.93923 to 0.94101, saving model to results/sentiment_imdb/lstm.h5\n",
      "782/782 [==============================] - 271s 346ms/step - loss: 0.2098 - accuracy: 0.9177 - AUC: 0.9728 - val_loss: 0.3557 - val_accuracy: 0.8515 - val_AUC: 0.9410\n",
      "Epoch 5/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.1830 - accuracy: 0.9298 - AUC: 0.9789\n",
      "Epoch 00005: val_AUC did not improve from 0.94101\n",
      "782/782 [==============================] - 201s 258ms/step - loss: 0.1830 - accuracy: 0.9298 - AUC: 0.9789 - val_loss: 0.3691 - val_accuracy: 0.8495 - val_AUC: 0.9392\n",
      "Epoch 6/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.1625 - accuracy: 0.9382 - AUC: 0.9832\n",
      "Epoch 00006: val_AUC did not improve from 0.94101\n",
      "782/782 [==============================] - 204s 261ms/step - loss: 0.1625 - accuracy: 0.9382 - AUC: 0.9832 - val_loss: 0.3621 - val_accuracy: 0.8562 - val_AUC: 0.9367\n",
      "Epoch 7/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.1459 - accuracy: 0.9453 - AUC: 0.9864\n",
      "Epoch 00007: val_AUC did not improve from 0.94101\n",
      "782/782 [==============================] - 211s 269ms/step - loss: 0.1459 - accuracy: 0.9453 - AUC: 0.9864 - val_loss: 0.3970 - val_accuracy: 0.8546 - val_AUC: 0.9327\n",
      "Epoch 8/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.1300 - accuracy: 0.9530 - AUC: 0.9889\n",
      "Epoch 00008: val_AUC did not improve from 0.94101\n",
      "782/782 [==============================] - 206s 264ms/step - loss: 0.1300 - accuracy: 0.9530 - AUC: 0.9889 - val_loss: 0.3789 - val_accuracy: 0.8561 - val_AUC: 0.9347\n",
      "Epoch 9/100\n",
      "782/782 [==============================] - ETA: 0s - loss: 0.1165 - accuracy: 0.9575 - AUC: 0.9911\n",
      "Epoch 00009: val_AUC did not improve from 0.94101\n",
      "782/782 [==============================] - 203s 260ms/step - loss: 0.1165 - accuracy: 0.9575 - AUC: 0.9911 - val_loss: 0.3858 - val_accuracy: 0.8509 - val_AUC: 0.9309\n"
     ]
    }
   ],
   "source": [
    "training = rnn.fit(X_train_padded,\n",
    "                   y_train,\n",
    "                   batch_size=32,\n",
    "                   epochs=100,\n",
    "                   validation_data=(X_test_padded, y_test),\n",
    "                   callbacks=[early_stopping, checkpointer],\n",
    "                   verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T01:37:03.545028Z",
     "start_time": "2020-06-22T01:37:03.533317Z"
    }
   },
   "outputs": [],
   "source": [
    "history = pd.DataFrame(training.history)\n",
    "history.index += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T01:37:04.464141Z",
     "start_time": "2020-06-22T01:37:03.561700Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+gAAAEYCAYAAADPrtzUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXxTVfr48U/SJF2S7gstFLoBLVSgogJVEEQZFQeEYhF1EOr2VZYZcHQEHbUiIjOOzjAuVUd0XBgFHZbhp+P3O0hxwYLK0MpSChQolK0rbdM0zXZ/f6QNLRYK2JI0fd6vV19J7pL7nATuuU/OueeoFEVREEIIIYQQQgghhFup3R2AEEIIIYQQQgghJEEXQgghhBBCCCE8giToQgghhBBCCCGEB5AEXQghhBBCCCGE8ACSoAshhBBCCCGEEB5AEnQhhBBCCCGEEMIDaNwdgBCi41mtVq677jpSUlJ466233B2OEEII0em6Ut23Y8cOfvOb37Bx48aLfo+6ujpmz57Ne++9B0BycjJ5eXmEhYVd9HsmJyfTv39/1Go1KpWKhoYGDAYD2dnZDBo0CACTycTLL7/Mxo0b0el0AIwdO5aHHnoIPz8/13utWbOGjz76CLPZjNVq5YorruDRRx8lKCiozWNf6PZCeCtpQRfCC/3nP/8hJSWFnTt3Ulxc7O5whBBCiE7X3eq+mpoaduzY0eHv++6777Ju3TrWrl3L//7v/zJ+/HgWL14MgM1mIysrC4fDwdq1a1m/fj2rVq2ivr6ee++9F5vNBsDrr7/Oxx9/zKuvvsq6detYt24dGo2GBx98sM1jXuj2QngzSdCF8EIffvgh119/PePHj+fdd991Lf/kk0+45ZZbmDBhAnfffTfHjx8/6/KtW7fyy1/+0rVvy9cvv/wy9957LxMmTOCRRx6hoqKCWbNmcfvttzN27FimT59OZWUlAAcPHmT69Omu9//ss8/Ytm0bY8aMweFwANDQ0EB6ejpVVVWX6iMSQgjhZdqq+7Zu3crEiROZNm0aEyZMwGKxsHHjRjIzM5k0aRLTpk1j+/btAOesy1p6+eWXWbRoUZuvp0+fzosvvshdd93F2LFjeeKJJ1x13T/+8Q9uvPFGpkyZwj/+8Y9W75mTk8PkyZO59dZbmTVrFidPnnS935w5cxg/fjzvv/9+q30WLlyI2Wzm1ltvxW63u2LJyMhg7NixrFixwrXtxx9/TEZGBpMmTWLmzJnn/QOGzWbj+PHjBAcHA/D555/jcDhYuHAh/v7+APj7+/PEE09gNBr5z3/+g8lk4o033mDJkiVEREQAoNVq+d3vfse0adOwWCytjnE+27f3mTd/Ru+++y7Dhw93HcNutzNq1CiKi4upq6tjwYIFZGRkMGHCBJYsWeL6QUEITyIJuhBeZv/+/Wzfvp2bbrqJSZMmsW7dOqqrq9mzZw9/+tOfeOutt1i/fj1jx44lJyfnrMvbc/ToUdasWcOf/vQnPv30U9LS0li5ciVffPEFfn5+rFu3DoCHH36Ym266iU8//ZQ333yTl156ieTkZIKDg/n6668B+PTTT0lPT/9Z3fKEEEJ0X2er+wD27dvHiy++yPr16zl27Bh//vOfefPNN1m7di3PPvssc+fOxWQynbMuuxCHDx/m/fff51//+hdfffUV3333HYWFhbzyyit88MEH/POf/0Sr1bq2X7t2LXv37uXjjz9m3bp1jB49mt///veu9UFBQXz22WdMnz691XGef/55V4w+Pj4A9O7dm9WrV/PKK6+wdOlSrFYr3333HWvXrmXFihWsXbuW++67jzlz5pw1/hkzZjBhwgRGjhzJjTfe6DoWwPbt27nyyit/so9KpSI9PZ1t27Zx4MAB/Pz8iI+Pb7WNv78/EydOdHWLb3ah27el+TOaMWMG/fr1c9068M033xAbG0tSUhJLliwhNTWV1atXs3btWqqrq3nnnXfafW8hLjW5B10IL/Phhx9y3XXXERoaSmhoKLGxsaxatQqdTsfIkSOJiYkBYObMmQC88847bS7funXrOY+TlpaGRuM8hcyYMYMffviBd955h0OHDrFv3z6GDBnCqVOn2LNnD5mZmQDExMSwYcMGAO666y5WrVrF6NGjWblyJb/73e86+qMQQgjRTZyt7ktLSyMmJoZevXoBsHnzZsrKylx1HTiTy8OHD5+1LrtQ1113HWq1GoPBQFxcHDU1NezevZtrrrmGyMhIAG6//Xa++eYbAHJzc9mxYwdTpkwBwOFw0NDQ4Hq/thLis2nu6TZgwAAsFgtGo5FNmzZRUlLCtGnTXNvV1tZy6tQpQkJCfvIe7777LmFhYezatYsHHniA4cOHEx4e7lp/tlZni8WCj48ParXa1WvgfFzo9m1p+RnddtttrFmzhptuuonVq1czdepUADZt2sSOHTv45JNPADCbzT/rmEJ0FknQhfAiJpOJdevWodPpGDt2LABGo5EPPviA++67D5VK5drWbDZz9OhRfHx82lyuUqlQFMW13Gq1tjpWQECA6/kLL7zAjz/+yJQpUxg+fDg2mw1FUVwJfMv3P3DgAD179mTChAm89NJLbNmyBZPJxFVXXdWxH4YQQohu4Vx132WXXdaqvnI4HKSnp/OXv/zFtez48eNERUWdtS47U3v1Y8uB0lpu23Kf5hbv5pjuu+8+7rzzTsCZ6NbU1LjWt4y/PWfWu4qi4HA4uPXWW3n00UddxysrK3N1Wz+b1NRUFi5cyIIFCxgwYACxsbEMHTqUt956C4fDgVp9uiOuw+Hg+++/56GHHqJv377YbDYOHTrUqlW8sbGROXPmsHjxYnr06OFafj7bX8g1yc0338zSpUspLi7m+++/Z+nSpa4Yly1bRlJSEuD8kaLl9YkQnkK6uAvhRdavX09ISAhff/01GzduZOPGjWzYsAGTyURdXR15eXmUlZUB8NFHH/HCCy8wfPjwNpeHhYVx7NgxKisrURSFTz/99KzH/eabb5gxYwaTJk0iPDycb7/9FrvdjsFgIDU1lbVr1wLOi6A77riDuro6V9e1xx9/vNWv+kIIIcSFOFfdd+bYJunp6WzevNl1D/aXX37JxIkTMZvNZ63LzhQaGsquXbtQFAWj0Uhubm67MV5zzTVs3ryZEydOAM4Ry5uNHDmSTz75BKPRCMCyZcvOq1eZRqPBbre3+SNCSyNHjuTTTz911fMffvghM2bMaPf9wdkiP3jwYFcX9xtvvBF/f3+WLFniaoE2m808++yz6PV6xo0bh06n4/777+eJJ56goqICcP7osGTJEhoaGlol58B5bX8hn7mvry+33HILCxYs4Be/+IXrXvmRI0fy97//HUVRsFgsPPTQQ3zwwQfn9TkIcSlJC7oQXuTDDz8kKyur1S/zQUFBTJ8+ndzcXB599FHuu+8+ACIjI1myZAk9evQ46/Jp06YxZcoUIiMjGTNmzFlHi509ezZ//OMfWbZsGVqtlqFDh3L48GEAXnzxRZ555hnef/99VCoVzz33nKuLX0ZGBqtWrWLSpEmd+bEIIYTwYueq+/7+97+32rZv374sWrSIhx9+2NXTKycnB71ef866rKWJEyfy9ddf84tf/IIePXowbNiwdpPk5ORkHn30UWbMmIFer2fw4MGudZmZmZw8eZKpU6eiUqmIiYlxtfqeS2RkJIMHD+aWW25pNSDcmUaOHMn999/PPffcg0qlwmAw8Morr5x36/GTTz7pKvOoUaN4++23ee2118jIyECtVmO32xk7dixvv/226976Bx98EH9/f+69917A2Ro+bNgwXnvttTaP0d72F/qZZ2Zm8sEHH5Cdne1a9sQTT/Dcc88xYcIErFYrV199tevaRwhPolLaO6MIIUQnUBSFv/3tbxw9epRnnnnG3eEIIYQQQgjhdtKCLoRwi+uvv56oqKiz/pouhBBCCCFEdyMt6EIIIYQQQgghhAeQQeKEEEIIIYQQQggPIAm6EEIIIYQQQgjhATw6Qf/xxx/dHUKnO3TokLtD6HRSRu/QHco4b948d4fQ6brD99gdyujppP72Dt2hjHLe9w5SRu/QHcp4Pjw6QW9r7klv09DQ4O4QOp2U0Tt0hzL+4x//cHcIna47fI/doYyeTupv79Adyijnfe8gZfQO3aGM58OjE3QhhLiUvvzyS3eHIIQQ4hKS874QwtNIgi6EEE1eeeUVd4cghBDiEpLzvhDC00iCLoQQTWROdiGE6F7kvC+E8DQadwdwoaxWK6WlpZjNZneH0iGsViuFhYWX5Fh+fn7Exsai1WovyfGEEEKIZlJ/Xzypv4UQovvocgl6aWkpgYGBxMfHo1Kp3B3Oz9bQ0IC/v3+nH0dRFCorKyktLSUhIaHTjyeEEEK0JPX3xZH6Wwghupcu18XdbDYTHh7uFZX7paRSqQgPD/ealgshOsPHH3/s7hCE8FpSf18cqb87l5z3hRCepssl6IBU7hdJPjchhBDuJPXQxZHPTQghuo8umaALIURnyMzMdHcIQpyVoijUNFgpLje6OxQhvIac94UQnUlRFIyNNo6eamDfybrz2qfL3YPuCZYuXcquXbsoLy/HbDbTu3dvQkND+etf/3rO/d58801GjBjB4MGD21z/3HPPkZWVRc+ePTsjbCGEEB6owWKnwthIWV0j5XWNlBudjxVNj+UtlltsDgAOLb3FzVF3TVJ/CyGEuFCKolBvsVPTYKXGZHU+NlipbTj9/Gx/tQ1WbA7F9V7nU39Lgn4RFixYAMDq1as5cOAAjzzyyHnt98ADD5xz/RNPPPGzYxNCCOF+VruDSqOlKbE2u5LsiuZlLRJxY6PtJ/urVBCu1xFh8CUy0JfECD2Rgc7nEQZfN5TIO0j9LYQQ3VNHJtlnUqsgyF9LcIu/XqH+rV4H+2sJDTi/mTi6dIL+z22lrPrhSIe+59QrezPlitgL3m/BggWcOnWKU6dOkZOTw5/+9CdOnDhBdXU11157LfPmzWPBggWMHz+eiooKvvzyS8xmMyUlJTzwwANkZGQwffp0srOz+eyzzygtLaWyspJjx46xcOFCRo0aRW5uLn/9618xGAwEBweTnJzM3LlzO7T8QnRns2bNcncIwoM5HArVJosrsT6ddLdOuMvrGqk2Wdt8jyA/DRGBvkQafEntGeRKuiMNvq7lUYG+hOl1aHy89y40qb+l/vYUct4XouvwhCQ7pOnRtW2AFoNOg1rdcWOFdOkE3dOMGDGCmTNnUlpaSlpaGpmZmTQ2Nroq+JaMRiPLly9nz549zJs3j4yMjFbrdTodb731Fps3b+btt9/m6quvZvHixaxcuZKIiAh++9vfXsqiCdEtzJkzx90hiEtMURTqGm2tu5KfJemurLdgb6Ny99OqXUl2QoSeYQlhRBr8mlq7da1avv20Pm4opWiP1N/dl5z3hbj0HA6FOrONKpOFqnoL1fUWqkwW9h46hd/hok5JskPOWN7ZSfbP0aUT9ClXxF7Ur+WdpXl+0pCQEHbs2MGWLVswGAxYLJafbJuSkgJAdHR0m+sHDBjQan1VVRUGg4GIiAgArrzySioqKjqrKEJ0S6NHj6asrMzdYYgO0NZ93RVnJNzNiXhj033dLWnUKlf38h5BflzWM7hVou1q+Q70Ra/zkVG2L5DU31J/ewo57wvx8zS3alfXO5PtKpPF9bzaZKGq3upKwKubllWbrG3+4A2gVlW5EueQpiQ6to2WbE9Psn+OLp2ge5rmC7TVq1cTGBjIokWLKCkpYdWqVSiK0ua27b1Xs/DwcOrr66mqqiIsLIyCggJ69erVsQUQopsrLy93dwjiApitdg5W1LO/zOj8KzdSXGbkcKURk/XAT7ZXqSAs4HSL9pn3dbfsbh7sr/WKSl6cH6m/uy857wvRmtlqdybargS7uYW7daLdvL663orF/tMfugF81CpCA3SE6bWEBujoG2UgVK8jLEDnfGxaHqbXERqg4+SRgwwdNLDb17+SoHeC9PR0Hn74YbZt24a/vz9xcXE/+9dZtVrNk08+yf33309gYCAOh4O4uLgOilgIITxXrdnqSsKLWyTjR6pMNP8Ar1JB79AA+kYZSA5VkRwXczrp7ib3dYufT+pvIYQ3abTZOWWytupGXt2caLdMwJsS7ap6Cw1We5vvpVJBiL/WlWD3DgtgSGzITxPtFgl4oO+FtWgbT6q7fXIO55GgOxwOsrOzKSoqQqfTsXjx4lYVy9q1a1m+fDmBgYFMnjzZNZ/kpEmTCAwMBCA2Npbnn3+ekpISFixYgEqlol+/fjz99NOo1V33YqnlfWdLly51Pe/Xrx/r16//yfYtt2nm6+vLxo0bAXj//fcBWg0ck5SU5Fq+Z88ePvzwQ3Q6HY888ggxMTEdUxAhBAADBw50dwjdlqIolBsbWyXh+5oey+oaXdvpfNQkROi5rGcwt6b1om+Ugb6RBhIj9a77uwsLCxkwoK+7iiK6AKm/RTM574uuwu5QqDA2/rT7+Bn3cZ9+tLY5S0izQD+Nq+U6KtCP5B5BzkS7VQu3zpV4B/tr8ZHk+ZJoN0HfsGEDFouFlStXkp+fz9KlS8nJyQGgqqqKZcuWsWbNGoKCgpg5cybp6elERkYCpyusZs8//zzz5s1j+PDhPPXUU3zxxReMGzeuE4rlnfR6PVOnTsXPz49evXoxfvx4d4ckhFf55JNP3B2C17M7FI5WN7C/vO501/Smv1rz6QsJg6+GpCgDo/pFOpPwpr/eof7SCi66HKm/PZec94W7NVjslNWZKatrpKy2kZO1Tc/rnFN0ltU6nztnBznY5nsE6HxatWAnROjPmmiH6rWE+OvQaaQu9VTtJujbtm1j1KhRAKSlpbFz507XutLSUlJSUggJCQFg0KBBFBQUEBsbS0NDA/fccw82m42HH36YtLQ0du3axbBhwwC49tpr2bx58zkTdIfDQWFhYatlVquVhoaGCy+ph1IU5bzLM2XKFKZMmeJ6bbfbL/izsFqtP/lMO5vZbL7kx7zUpIze4YknnuC5555zdxid6lJ9jxa7wrFaK0dqLByusXLklIUjNVZKa61Y7Kfv6Q3x86F3sJaRffzpE6Kjd7CW3sE6IgJaDrxmAaowl1ex7zxuF+0O/1abByITXcOvfvUrfvWrX7k7DNGGp59+mlWrVrk7DOFlFEXB2GijrM6ZcLdMtF2JeJ2Z8tpG6tpo5db6qIg0+BIZ5Eef8ACujA8Fcy3943q1SLy1rsRbZgjxLu0m6EajEYPB4Hrt4+ODzWZDo9EQFxfH/v37qaioQK/Xk5eXR3x8PH5+ftx7771kZmZy6NAh7r//fj7//HMURXFdcOn1eurq6s55bLVa/ZOLkMLCQvz9/S+mrB6poaHhkpZHq9Ve8gs7Z3dT776YlDJ6hzVr1rB69Wp3h9GpOvp7NDbaftISXlxu5HCVqdUIrbGh/vSNCmZsqqFVi3hIgK7DYmnWHf6tCiE6xscff+zuEEQXoigKp0xWVwu3M+lu+fx0At7Wvdy+GjVRQb5NXcoDubZfJJGBznFSooL8iGqaOSSkjYFKnXVb/CUqqXCndhN0g8FAfX2967XD4UCjce4WHBzMwoULmTt3LtHR0aSmphIaGkpCQgJxcXGoVCoSEhIICQmhvLy81f3m9fX1BAUFdUKRhBBCdCRFUaist/wkCd9fZuR4jdm1ndZHRXy4npToQH45OIa+UQaSIp1//jr5dV8IIYRncjic9Vxzgl3elGyfPCPpLjc2Ymljak69zseVYA+ODXEm3IG+rmS8R5AvkYF+BPlpZFpO0a52E/ShQ4eSm5vL+PHjyc/Pp3///q51NpuNgoICVqxYgc1mIysri/nz5/PJJ5+wd+9esrOzOXnyJEajkcjISAYOHMjWrVsZPnw4X331FSNGjOjUwgkhhDh/DofC0VMNrunKWk5fdspkdW0XoPMhKdLAiMTwVq3hfcIC0Mr94UIIITyEze6gwmhxtXCfbNHqXX5G4t3WvNxBfhp6BPkRFeTLsIQwopqm4+zRlIw3J+V6X5kYS3Scdv81jRs3js2bNzNt2jQURWHJkiWsX78ek8nE7bffjlarJSMjA19fX7KysggLC+O2225j4cKF3HHHHahUKpYsWYJGo+Gxxx7jySef5KWXXiIxMZEbb7zxUpRRCCHOy6ZNm9wdwiVhsTkoqWw9f/j+MiMHyutbdckL0+voG2ng5stiWiXiMUF+Mg2KEMIrdJfzvjcyWWwcqWrgcJWJ7UU1fHq4qFVrd1mdmcp6C8pP827C9Tpn1/IgP/r3CHR1LW/Z6h0Z6Cv3dgu3aDdBV6vVLFq0qNWypKQk1/M5c+YwZ86cVut1Oh0vvvjiT94rISGBDz744GJj9Qh33XUXc+bMIT093bVs8eLFJCcnu6aYazZ27Fj+/e9/8+677zJixAgGDx7sWtfY2MjNN9/Mp59+etZjrVy5koyMDPbv388XX3zxk89ZCNGxdu3axejRo90dRodQFIWyukYOlNdzqLKegxXOvz1HqzhedxBbi5aCXiH+JEUZGJ7QukU8TN/x94cL4S5Sf4u2eNN539vYHQonas0crjRxpMrE4SoTR6qbHqtMVBgtrbZXqyqJMDgT7OhgPwbHBrtauFu2dkcYfGUEc+HRpD/GBZo6dSrr1q1zVfAWi4Xc3Fwefvjhs+7zwAMPXNSx3njjDSZNmsSAAQNkwCMhLoHZs2cza9Ysd4dx3hRFodpk5WCFkYMVJg5WGDlUYeJART0llfWYLKdbw3UaNXFhAcQGaZlweR/6RhnoFxVIYqReuuaJbkHqb9GWrnbe9zY1DdbTyXfTY/Pzo6casLaY9UOtgp4h/vQJC+CGAT3oHRZA77AA+oQFUF92hBGXXybzdAuv0LWvyvI/hO0d3CJ/+a8g7Y6zrr7pppv4y1/+4hp9/YsvvmDEiBE8/PDDNDY2curUKWbPns0NN9zg2mfBggWMHz+eK664gkceeYTa2lr69OnjWv/dd9/xyiuvAM7pgf7whz/www8/UF5ezvz585kxYwYfffQRf/7zn/nXv/7Fu+++i06nIz4+nkWLFrF+/Xq+/PJLzGYzhw8f5v777ycjI6NjPxchhNvUmq0cqjjdCn6oop6DlSYOlreeO9xHraJ3qD/xEXpGJIaREKEnIUJPfLieniH++KhVTaPAprixNEIg9bfU36KbsNgcHDvV4Gr5bpmIH6lqoKbB2mr7kAAtfcICSO0VzE2XxdCnKQHvExZATIjfWcc5KTQel+RceI2unaC7ga+vL9dffz3/+c9/mDhxIqtXr2bYsGFMnDiR4cOH89///peXX365VQXfbM2aNfTv35/58+dTUFDA1q1bAdi3bx8vvPACPXr04PXXX+fzzz/noYceIicnhz//+c/k5+cDUF1dzcsvv8yaNWswGAwsWbKElStXEhAQgNFoZPny5Rw6dIgHH3xQKnghupgGi51Dlc7k+0BTEt7cNb1lNz6VCnoG+xMfEcDEtJ7Eh+tJjHQm4b1lkDbRDofDQXZ2NkVFReh0OhYvXkxcXJxr/dq1a1m+fDmBgYFMnjyZzMxMrFYrCxYs4OjRo6jVap599tlWt7p1FVJ/C9Hxmmf5OLMVvPne8OM1DbQce03ro6J3aACxYQGk9Q5xJd/NreFBflr3FUYID9G1E/S0O875a3lnyczM5I9//CPDhw+ntraWMWPGkJOTwyeffIJKpcJms7W53759+xg1ahQAQ4YMcU1X16NHD5577jkCAgI4efIkQ4cObXP/I0eO0LdvX9e89FdddRXffPMNQ4YMISXF2SIWExODxWJpc38hxLllZ2d36vtbbA4OV5lcyXdzIn6wor7VdGUAkYG+JITruT6lB/FNLeEJEXriwgNk0Bpx0TZs2IDFYmHlypXk5+ezdOlScnJyAKiqqmLZsmWsWbOGoKAgZs6cSXp6Onv27MFms/HRRx+xefNm/vKXv/Dyyy//vECk/pb620N09nnfG5itdkqbW8ArTRyucraINyfjLW+nAmf91TvUn6viQ+kT1qtVV/QeQX7S0i1EO7p2gu4mycnJ1NfX89577zFlyhSWLVtGZmYmo0eP5p///Cdr1qxpc7/ExETy8/O54YYb2L17t+tC4Pe//z0bNmzAYDDw2GOPoTQNN6lSqXA4Ts+1GBsbS3FxMSaTiYCAAL777jsSEhJc2wohfp6pU6f+7PewOxSOVjdwsLK+Vbf0gxX1lFabWrUkhARoSYjQk54Y3ioJj4/QY5D7wkUn2LZtmyvRTEtLY+fOna51paWlpKSkEBISAsCgQYMoKCggJSUFu92Ow+HAaDS6ktOuSOpvcaaOOO93dQ6Hc1DRll3QW94PXlbX2Gp7P63a1fKdnhRO79CmbujhAcSG+hOg67rnCCE8gfwPukhTpkzhhRdeIDc3l4CAAJ577jneeOMNYmJiqK6ubnOfu+66yzX9XGJiIlqtsxvPrbfeytSpUwkKCiIiIoKysjIArrzySh544AFmz54NQFhYGHPnzuXuu+9GrVbTp08fHnnkkXOOJCuEOH8DBw50XWCfi6I4R5Z13g9uajVI25GqBiz20xfmep0PCZF6BscGc2taT1cCnhCuJ1RGSReXmNFodLXiAvj4+GCz2dBoNMTFxbF//34qKirQ6/Xk5eURHx9PQEAAR48e5eabb6a6uprXX3+93eM4HA4KCwtbLbNarTQ0NHR4mS7UxIkT+fOf/8znn3+ORqPh2WefJScnhx49elBZWUlDQwMOh4OGhgZsNhsWi4WMjAyefvppbr/9dhISEtBoNCiKwi233EJmZiaBgYGEh4ejUqloaGggLS2Ne++9lwcffBC73Y6/vz//8z//w69+9SvUajW9e/dm9uzZfP7559hsNhoaGmhsbHQdty1Wq/Unn2lnM5vNl/yYl9rAgQPZvXu3u8PoVGazmW0/7uJEnZUTRpvzsc7GcaOVk0YbJ+psWFv8eqwCIvQ+RBu0DInSEp0UQHSghhiDluhADSF+Pmf8sGQGzNirqiiputSla4qgG/xblTJ6h/MZOFSlnM/VqJts376dyy+/vNUy5wBH3jMiavNgNZeKOz4/b/vO2iJl9A4qlcqVoCuKQlW9pVUL+KHKeg6U11NSaVc8WS8AACAASURBVGo1X7hOoyYhXE98RADxEXoSmwZmS4jUE2nw9agWsu7wPXaHMl6s559/niFDhjB+/HgArr32Wr766ivX+o0bN/K3v/2N6OhoDAYDo0eP5vvvv0en0/Hb3/6W48ePM2PGDNavX4+vr+9ZjyP1d8eT+rtztDzve4NGm519J43sOlbD7mO17D5eS9HxGmobHa22C/TVuLqd9wk/3QW9d6g/vUL98dV0rVupusO/VSlj9yEt6EKIbs1md3Cwop6ik3UA/Oaj7a6B2upajJCuUavoE+ZMwK/pG+FqBU+I1BMT5Ida7qkTXcDQoUPJzc1l/Pjx5Ofn079/f9c6m81GQUEBK1aswGazkZWVxfz58ykqKnL1+AoODsZms2G32892CCHEJVJjsrL7uDMJb07I95cZsTW1hut1PgyICeLqPnqGJPWid5i/q2t6sL/Wo348FkKcJgm6EKJbUBSFYzVm9p6oY8+JOopO1FJ00khxmdHVJd0/6Sq2lVSTEKFn8uW9nK3gTfeF9wr1lxHSRZc3btw4Nm/ezLRp01AUhSVLlrB+/XpMJhO33347Wq2WjIwMfH19ycrKIiwsjJkzZ/L4449z5513YrVamT9/PgEBAe4uihAdYsyYMe4OoV2KonC8xsyuY7XsPtaUjB+vpbT69O0QkYG+pPYMYmxKFAN7BpHaM5i4sADUruk1u97MC0J0V10yQVcURX71uwje1IVLiHM5ZbJQdKKOopPOZHxv0/OWLeIxwX4kRwdybf8IknsEkhwdiO1X7zDkslQ3Ri5E51Kr1SxatKjVspZTps2ZM4c5c+a0Wq/X61m2bFmHHF/q74sj9Xfnee2119wdQis2u4MDFfWtEvHdx2qpNjnnC1epICFcz5DeIdw5vA8DY4IY2DOIqEA/N0cuhOgoXS5B9/Pzo7Ky0jUYizg/iqJQWVmJn5+cwIX3MFvt7C8ztmoRLzpRy8na0yPOBvlpSIkOYlJaL/pHB5ISHUj/HoEE+/90rtXr7ppEbm7upSyCEN2G1N8XR+rvzjVr1iy3nfdNFhuFx+tcSfjuYzXsOVFHo83Zq0unUZMSHciNqdFNreJBJEcHySwfQni5Lvc/PDY2ltLSUsrLy90dSoewWq2ue/s6m5+fH7GxsZfkWEJ0JLtDoaSynqKm7ul7T9ZRdKKOQ5X1rmnLdBo1/aIMXNM3wpWEp0QH0SPo/Adp27RpU+cVQohuTurviyf1d+e5VOf9CmNjU6v46XvGD1bU09w5Ithfy8CYIKaPiHN1UU+M1MutVUJ0Q10uQddqta65Q72BjFYoxGmK4pyL1dUifsJI0cla9p00uloUVCqID9eT3COQCUN6khzt7J4eFxaARi5khPBYUn+L7sDhUDhSbfrJ/eIte3b1CvFnYM8gJgzuSWpPZxf1XiH+0rNECAF0wQRdCOEdas1W14Bte0+efjzVdJ8dQFSgL8nRgdydHudqEe8bZcBf17WmfxFCCOF9mqc0O91F3dk6bmx0jnfio1bRN9LANUkRDGxKxAfGBBESoHNz5EIITyYJuhCiUzXa7BSX1VN0sqlF/EQtRSfqOFZjdm0T6Kuhf3Qg4wfFuAZsS+4RSKj+0l7E7N69+5IeTwghhHud73m/1mx1JeHN3dT3l9VhtTv7qAc0TWk2+fJerlbx/j0C8dPKD8pCiAsjCboQokM0d+trHjV9T9N94gcr6rE33Siu9VGRFGlgWEIYydFBJEcbSI4Oomewn0d07Vu1ahVPP/20u8MQQghxiZx53lcUhRO15tOJ+LFadh2v4UjV6SnNIgzOKc3GJEc6k/GYIOLC9fio3V+PCSG6PknQhRAXrLyu0TWNWXOL+N6TRhqsdtc2fcICSI4O5ObLopu6pwcSH+HZA95kZ2dLgi6EEN2E3aGQnZ1N2q33uZLx3cdrqaq3uLZJiNAzODaEaVf1cY2kLlOaCSE6kyToQohzKqs1k3egko0FFZR9vYW9J+uobHHxEmHQkRwdyB3D+rhaxPtFGdDLNDBCCCE8iKIo7D1p5NviCr4trmTLgUoAfvNRPjofNcnRgYwb0IPUXs5W8ZQYmdJMCHHpyVlHCNFKWZ2ZrQeqyDvgvHg5UF4PgJ9GRUpMMOMG9nC1iPePDiTC4OvmiIUQQoifUhSFw1Umvi2u5NviSvKKK6gwOn9g7hMWwC2DYtgJfD5vFEmRBo/u4SWE6D4kQReim6swNrKlKRnfcqCK/WVGwDlw27CEMO64qg8jEsNR1RzlstSBbo62c7366qvuDkEIIcTPcKLG7Gohzyuu5Ogp573jUYG+jOoXSXpSOOmJ4fQOCwCgz6uvkhId5M6QhRCiFUnQhehmKo2NbD1YxZYDzouXfU0JucFXw1XxoWReEUt6UjgDY4JazSteWHfMXSFfMqmpqe4OQQghxAWoNDay5UAV3xZXkFdcyYEKZ6+vkAAt6YnhPDgmiauTwkmM0Lc5GKmc94UQnqbdBN3hcJCdnU1RURE6nY7FixcTFxfnWr927VqWL19OYGAgkydPJjMzE6vVyuOPP87Ro0exWCw89NBDXH/99ezatYsHH3yQ+Ph4AO644w7Gjx/faYUTQkBVvYXvDjpbx/OKKyk6WQc4p4S5Kj6MjKHOhPyynq0T8u5ozJgxKIri7jCEEEKcRZ3ZyncHq/i2uJLN+yvYc8JZp+l1PgxPDOfO4X1ITwpnQHQQ6vMYVV3O+0IIT9Nugr5hwwYsFgsrV64kPz+fpUuXkpOTA0BVVRXLli1jzZo1BAUFMXPmTNLT09m6dSshISG88MILVFdXM3nyZK6//np2795NVlYW99xzT6cXTIju6pTJwtaDzmR8y4FK18WLv9aHK+NDufXynoxIDGdQr2C5304IIYRHa7DY2VZS7eq2vuNoDXaHgq9GzZXxoTx6YzLpSVKnCSG8R7sJ+rZt2xg1ahQAaWlp7Ny507WutLSUlJQUQkJCABg0aBAFBQXcdNNN3Hjjja7tfHx8ANi5cycHDx7kiy++IC4ujscffxyDwdChBRKiu6kxWdna1EK+5UAlhSdqURTw06q5Kj6MCUN6MiIxjMGxIXLxIoQQwqNZbA5+LD3F5v2VfFtcwfbDp7DYHWjUKob0DmHWmCTSk8IZ2icUP62Pu8MVQogO126CbjQaWyXRPj4+2Gw2NBoNcXFx7N+/n4qKCvR6PXl5ecTHx6PX6137/vrXv2bevHkADB48mMzMTC677DJycnJ49dVXeeyxx856bIfDQWFh4c8to0czm81SRi9wKctYb3Gw42QDO06YKTjRwIEqCwqg81ExMMqP6UNCGRzjT/9wX7Q+KsAKppPs33vyZx23O3yPkydP9voydofvsTuUccCAAe4OQYgOYXco7D5W62oh//5QFSaLHZUKUnsGMfOaeNKTwrkqPqxTpjzLzMzs8PcUQoifo90zncFgoL6+3vXa4XCg0Th3Cw4OZuHChcydO5fo6GhSU1MJDQ0F4Pjx48yePZs777yTCRMmADBu3DiCgoJcz5999tlzHlutVnv9RUhhYaGU0Qt0ZhlrzVZ+OFTluod817EaHAroNGqu6BPKxKHhpCeFM6R3ML6azmtN6A7f43PPPef1ZewO32N3KKMQXZWiKOwrM/Lt/tNzkdeabQD0izI0DVQawYjEMEICdJ0ezzPPPNPpxxBCiAvRboI+dOhQcnNzGT9+PPn5+fTv39+1zmazUVBQwIoVK7DZbGRlZTF//nwqKiq45557eOqpp0hPT3dtf++99/Lkk08yePBg8vLyZORMIdpgbLTx/aEqtjRduOw42pSQ+6i5vE8Iv76+HyMSw0nrHSLd+zrYbbfdxq5du9wdhhBCeA1FUThS1cDmFlOfVRgbAegd5s/Nl8VwdV/n1GdRQX6XPD457wshPE27Cfq4cePYvHkz06ZNQ1EUlixZwvr16zGZTNx+++1otVoyMjLw9fUlKyuLsLAwFi9eTG1tLa+99hqvvfYaAH/729/Izs7m2WefRavVEhER0W4LuhDdQX2jjR9Kql2DujUPgKPzUZPWJ4Q5Y/sxIjFM7re7BHbv3u3uEIQQoss7UWMm70AF3+6v5NsWc5FHBvoysm84VydFkJ50ei5yd5LzvhDC07SboKvVahYtWtRqWVJSkuv5nDlzmDNnTqv1v//97/n973//k/dKTU3lo48+uthYhfAKJouNHw5VO+chP1DJjtIabA4FrY+KtOYBcBLDubxPKP46SciFEEJ4tqp6C1sOVLruIz9Q7rw1MtjfORf5/4xO5OqkcJIiDW3ORS6EEOK0jh9tQwjRSvMUMc0JecGRU9gcimtE2gdHJzEiMZwr4iQhd7fIyEh3hyCEEB6vzmzl+0NVTSOtV1J4vBZwzkU+LCGMO65yzkU+MOb85iJ3JznvCyE8jSToQnQws9XOf0uqyTvg7LKef+QUVruCj1rF4NhgHrg2kRGJ4VwZH0qATv4LepIvv/zS3SEIIYTHMVtbz0X+Y2nTrVgaNVfGhfLIL/qTnhTB4NiuNxe5nPeFEJ5GsgMhfqZGm50fTzTw79K9zoS8ac5WH7WKQb2CuXdkIulJ4VwZF4q+E6aIER3nlVde4dVXX3V3GEII4Xb7y4z8o6CafV/n8d+S0/XakNhgHhqdxNVJ4QyN6/pjo8h5XwjhaSRbEOIilFab2FRUzqaicr4trsBksaNWwaBewWRdE8+IpoQ80E/r7lDFBXjttdfkQk0I0a1tP1xNzqZi/m/3SVTAwJ5BzLg6jquTIrgqoXPmIncnOe8LITyNd51lhegkFpuDH0qqmpLyMvaeNAIQG+rPlKGxJAaYmXLtEIIkIRdCCNHFKIrC1/sqyNlUTN6BSoL8NPx6bF9GRFi4euggd4cnhBDdiiToQpzFiRozm4rKyC0q45t9FdRb7Gh9VAxPCGfqlb0ZkxxFUqQelUpFYWGhJOdCCCG6FLtD4fOdJ8j5cj87j9bSI8iXJ8YP4I7hfTD4aigsLHR3iEII0e1Igi5EE6vdwX9LqsltaiXfc6IOgF4h/tx6eS+uS47i6qRwuY/ci3388cfuDkEIITpdo83O6v8e5Y0vizlUaSIhQs8fpgxi0uW98NV07XvKL5Sc94UQnkYyDdGtldWand3W95bx9d4K6hptaNQqrooP4/HxKYxJjqJflMzbKoQQouszNtr4x9YS3vr6IGV1jQzqFcxrdw3lxtRofDx8OjQhhOguJEEX3YrN7iD/yClyi8rYVFTOrmPOuVujg/y4ZXAMY5KjuKZvuAzu1k1lZmaiKIq7wxCi0zgcDrKzsykqKkKn07F48WLi4uJc69euXcvy5csJDAxk8uTJZGZmAvDGG2+wceNGrFYrd9xxh2u56BoqjY28s/kQ7+UdotZs45q+4bw0NY1r+oZ3+x+g5bwvhPA0kqALr1de18hXe8vJLSrj630V1DRY8VGruCIulN/dlMx1yVGkRAd2+4sUIYT327BhAxaLhZUrV5Kfn8/SpUvJyckBoKqqimXLlrFmzRqCgoKYOXMm6enpHD16lO3bt/Phhx/S0NDA22+/7eZSiPN1pMrEW18fYOUPR2i0ObhxYDQPjkkirXeIu0MTQghxFpKgC69jdygUlJ5i054yNu0t58fSGgAiA335xcAeXJcSxTV9Iwj2l1ZyIUT3sm3bNkaNGgVAWloaO3fudK0rLS0lJSWFkBBn8jZo0CAKCgrYs2cP/fv3Z/bs2RiNRn73u9+5JXZx/opO1PH6l8X8q+AYKmDy5b34n9GJ9I0KdHdoQggh2iEJuvAKVfUWVyv5V3vLqTZZUatgaJ9QHvlFf8YkRzEwJgi13GMnzmHWrFnuDkGITmU0GjEYDK7XPj4+2Gw2NBoNcXFx7N+/n4qKCvR6PXl5ecTHx1NdXc2xY8d4/fXXKS0t5aGHHuLzzz8/Z68jh8Ph9SOAm81mjytjYZmZlTtOsbXUhK9GxcSUICYPDCZSr8FaWUph5YW9nyeWsaM98MADXl/G7vA9Shm9Q3co44ABA9rdRhJ00SU5HAo7jta47iUvKD2FokC4Xsd1KVGMSY7i2n4RhATo3B2q6ELmzJnj7hCE6FQGg4H6+nrXa4fDgUbjvBQIDg5m4cKFzJ07l+joaFJTUwkNDSUkJITExER0Oh2JiYn4+vpSVVVFeHj4WY+jVqvP6yKkKyssLPSIMiqKwqa95eRsKua7g1WEBGiZd0M/ZqTHE6r/eXWgp5SxM82bN8/ry9gdvkcpo3foDmU8H5Kgiy7jlMnCV/sq2LSnjC/3llNZb0GlgiGxIcy7vj9jkiMZ1CtYWsnFRRs9ejRlZWXuDkOITjN06FByc3MZP348+fn59O/f37XOZrNRUFDAihUrsNlsZGVlMX/+fHx8fHjvvffIysqirKyMhoYGVzd44T42u4PPdp4gZ1MxhcdriQn246lfDmTasN4E6OTy7nzJeV8I4WnkDC48lsOhsPt4LZuKysgtKmf74WocCoQGaLm2fyTXJUdxbf9Iwn5mC4EQzcrLy90dghCdaty4cWzevJlp06ahKApLlixh/fr1mEwmbr/9drRaLRkZGfj6+pKVlUVYWBjXXXcd33//PbfddhuKovDUU0/h49O95sr2JGarnU+2lfLmVwc4XGUiKVLPC7cN5ta0Xug0aneH1+XIeV8I4WkkQRcepabByjf7KsgtcraSl9c1AjA4Npg5Y/sxJjmSIbEhMl+rEEJcBLVazaJFi1otS0pKcj2fM2dOm7d6yMBw7ldrtrJiy2GWf3OQCmMjQ2KDeXz8FfxiYA/pOSaEEF5EEnThVoqiUHi8jk17y9i0p5xth6uxOxSC/bWM6hfhaiWPDPR1d6iiGxg4cKC7QxBCiFbK6xp5Z/NB3s8roa7Rxqh+ETw0Jo30RJnDvCPIeV8I4WkkQReXXJ3Zyub9FeTuKefLveWcqDUDkNoziAdHJ3JdchRpvUPQ+EhXPXFpffLJJ+4OQQghADhcaeLNr4tZ9UMpVruD8ZfF8ODoJAbFBrs7NK8i530hhKeRBF10OkVRKDpR13QveRk/HKrG5lAI9NUwqn8EY5KjGNM/kqggP3eHKrq5p59+mlWrVrk7DCFEN1Z4vJbXvyxmfcExfNQqpgyN5YFrE0mMNLS/s7hgct4XQngaSdBFpzle08CKLYdZ9d0RyuoPApASHch9oxK5LjmSoXGhaKWVXHiQjz/+2N0hCCG6qe8PVfFa7n5yi8rR63y4b1Qi91yTQHSw/HjdmeS8L4TwNJKgiw6lKApbD1bxXt4h/nfXSRyKwpU9/Zl/4wDGJEcSE+zv7hCFEEIIj6AoChv3lJGzqZgfSqoJ0+v47bj+3J0eT3CA1t3hCSGEcANJ0EWHMFlsrN1+jPfyDrHnRB3B/lruG5nAr0bEYTxZwoABfdwdohBCCOERbHYH/+/H4+RsKqboZB29Qvx5ZmIqU6/sjb9OprATQojurN0E3eFwkJ2dTVFRETqdjsWLFxMXF+dav3btWpYvX05gYCCTJ08mMzPzrPuUlJSwYMECVCoV/fr14+mnn0atli7OXVlJZT3v55Ww6ocj1JptDIgJ4g9TBjFxSC/XRUbhSTcHKcR52rRpk7tDEEJ4MbPVzqofjvDmVwcorW6gX5SBl6YOYcKQnnLLl5vIeV8I4WnaTdA3bNiAxWJh5cqV5Ofns3TpUnJycgCoqqpi2bJlrFmzhqCgIGbOnEl6ejq7d+9uc5/nn3+eefPmMXz4cJ566im++OILxo0b1+mFFB3L4VD4al857+WVkFtUho9KxU2XRTPj6niujAuVaV9El7Vr1y5Gjx7t7jCEEF6mpsHKB1tKePubg1TWW7i8TwhPT0jl+pQomcPczeS8L4TwNO0m6Nu2bWPUqFEApKWlsXPnTte60tJSUlJSCAkJAWDQoEEUFBTw448/trnPrl27GDZsGADXXnstmzdvlgS9C6k1W/nkh1Le31LCwYp6Igy+zB3bjzuH9ZFBbIRXmD17NrNmzXJ3GEIIL1FWa2b55oOs2HIYY6ONMcmRPDQ6iWEJYfJjtoeQ874QwtO0m6AbjUYMhtNTe/j4+GCz2dBoNMTFxbF//34qKirQ6/Xk5eURHx9/1n0URXFVSHq9nrq6unMe2+FwUFhYeLFl6xLMZrPHl7HklIX1e2r5orgOs00hJdKX342K4po4PTofO9XHDlJ97Oz7d4Uy/lxSRu/h7WXsDt9jdyjjgAED3B2COIdDFfW88dUB/rmtFJvDwS2De/Lg6ERSe8oc5kIIIc6t3QTdYDBQX1/veu1wONBonLsFBwezcOFC5s6dS3R0NKmpqYSGhp51n5b3m9fX1xMUFHTOY6vVaq+/CCksLPTIMtrsDr7YU8a73x7i2+JKdBo1Ewb3YsbVcQyODbmg9/LUMnYkKaP38PYydofvsTuUUXimnUdreP3LYj7bcRyNj5rMK51zmMeF690dmhBCiC6i3QR96NCh5ObmMn78ePLz8+nfv79rnc1mo6CggBUrVmCz2cjKymL+/PnY7fY29xk4cCBbt25l+PDhfPXVV4wYMaLzSiYuSlW9hZXfH+GDLSUcPdVAz2A/Hr0xmWlX9Sbc4Ovu8IToVNnZ2e4OQQjRxTRPL/rapmK+2luOwVfDA9cmcc818UQFye1fnk7O+0IIT9Nugj5u3Dg2b97MtGnTUBSFJUuWsH79ekwmE7fffjtarZaMjAx8fX3JysoiLCyszX0AHnvsMZ588kleeuklEhMTufHGGzu9gOL87Dxaw7vfHmJdwTEsNgfpieE8+csB3DCgBxoZWVZ0E1OnTnV3CEKILsKhKPzfrhPkfFnM9sOniDDo+N1Nydw1PI5gf5nDvKuQ874QwtO0m6Cr1WoWLVrUallSUpLr+Zw5c5gzZ067+wAkJCTwwQcfXGysooNZbA7+vfM47+WVsK2kGn+tD5lXxHJ3ejzJ0YHuDk+IS27gwIEoiuLuMIQQHm774Wp+s66UwzUH6R3mz7OTLiPzilj8tDKHeVcj530hhKdpN0EX3qes1syKrYf5x3eHKa9rJD48gCd/OZDbroiVX/2FEEKIc1AUhSfX7aTe4mDZtDRuGRQjPc2EEEJ0GEnQuwlFUdhWUs27eSX8e8dxbA6F65IjufvqeEb3i5R5WIUQQojz8P2hanYerWXuiAhuTevl7nCEEEJ4GUnQvZzZaudf+cd4N+8Qu47VEuinYcbV8UwfEUd8hIwqK0RLY8aMcXcIQggP9/Y3BwkJ0DI2ydD+xsLjyXlfCOFpJEH3UqXVJt7fUsLK749wymSlfw8Dz02+jElpvdD7ytcuRFtee+01d4cghPBgR6pM/N/uEzw4Ogk/jdy37A3kvC+E8DSSqXkRRVH4triSv397iC8KT6JSqfjFwB7cnR7PiMQwVCrpxi7EucyaNYvc3Fx3hyGE8FB///YQapWKu9PjqT520N3hiA4g530hhKeRBN0LGBttrPlvKe/mlbC/zEiYXsdDY5K4a3gcPUP83R2eEF3Gpk2b3B2CEMJD1ZmtrPz+CLcMjiE62I/qY+6OSHQEOe8LITyNJOhdWHG5kffzSvhkWynGRhuDY4N5MXMItwyOkalehBBCiA708Q/OuvbekQnuDkUIIYQXkwS9i7E7FDYVlfH3bw/x9b4KtD4qbhkUw4yr40nrHSLd2IUQQogOZncovPPtQa6MC2VwbIi7wxFCCOHFJEHvImpMVlb9cIT3t5RwuMpEjyBffjuuP9OG9SEy0Nfd4QnhFXbv3u3uEIQQHug/u09ypKqBx28e4O5QRAeT874QwtOo3R2AOLfC47UsXP0jw5/fwHOfFRId5Merdw7lm8fGMvf6fpKcC9GBVq1a5e4QhBAe6O3NB+kV4s+4gT3cHYroYHLeF0J4GknQPZDV7uDTH48z9fU8bl72NWu2H2VSWi8++/UoVj2Yzi2DY9D6yFcnREfLzs52dwhCCA+z82gN3x2sIuuaeDRS93odOe8LITyNdHH3IOV1jXz03WFWbD3MiVozsaH+PD4+halX9iYkQOfu8IQQQohu5+1vDqLX+TD1qt7uDkUIIUQ3IAm6B9h+uJr38kr49MfjWOwORvWLYPGky7guJQoftQz6JoQQQrhDWa2Z9T8e467hcQT5ad0djhBCiG5AEnQ3OlJlYt6nRymqOIDBV8Odw/swPT2OpEiDu0MTolt69dVX3R2CEMKDvL+lBJtDIeuaeHeHIjqSzQKNddBYy+t/egaM5eAbCFo/d0cmhBCSoLvTH/+3iJJTFhbdmkrG0FgMvvJ1COFOqamp7g5BiE7lcDjIzs6mqKgInU7H4sWLiYuLc61fu3Yty5cvJzAwkMmTJ5OZmelaV1lZSUZGBm+//TZJSUnuCP+SMlvtrNh6mBsG9CAuXO/ucISigK3RlVi3fqw7y+umP3NN69f2RtfbTqhzwJ9edL7w0TkTdddf0BmPLZb7tbGseTuNL8i0t0KIiyQZoZvsLzPy/348xtTLQrg7Pd7d4QghgDFjxqAoirvDEKLTbNiwAYvFwsqVK8nPz2fp0qXk5OQAUFVVxbJly1izZg1BQUHMnDmT9PR0YmNjsVqtPPXUU/j5dZ8WxjXbj1JVb+HekQnuDqVrUxSwNrRIkGvaSKLbSKzNbSThDmv7x1Nrz0iegyCo5xnJ9Omku9fld6FsffP0ccxnxFJ7DBr3nF53vjGcM5k/I6Fv/vM7Y5nGTxJ9IbohSdDd5NXc/fhrfZg8MNjdoQghhOgmtm3bxqhRowBIS0tj586drnWlpaWkpKQQEhICwKBBgygoKCA2NpY//OEPTJs2jTfffPO8juNwOCgsLOz4AlwiiqKQ80UpiWE6gswnKSws+8k2ZrO5S5exXYqCpb6afdu+RG2rR22tx8fqfFRbTahtLV/Xo7Ya8bG1fF2Pj82E2lqPSrG3eziHjy8OTQAOrR67Vo9Dq8ehCcER3Au71uB8rdXjKJPbygAAIABJREFU0AScXq/V49DoW71WfC58+tnCwJEQeH7bquyW0+W11rs+m1afT6vPy4jaVI9PTTnqps9DbTWiPo9EX1H5NJXtdPntmtNltbf4DJyvW3xOLbY1WxXv/rdKN/j/iJTRWwwYMKDdbSRBd4MD5UbW5R/l/lGJBHefxgghhBBuZjQaMRhOj3Pi4+ODzWZDo9EQFxfH/v37qaioQK/Xk5eXR3x8PKtXryYsLIxRo0add4KuVqvP6yLEU321t5zDNQd5MXMIAwfGtrlNYWFhly6ji6UeKvc7/yr2Q+W+088tde3vrw1o3eobGAq+fdroFt6iq/iZLdw6A2qNzm1z/7rlezyzu36rlnvnc1VjLZq2uuvXHWuzu/7ZONRa1PoICAiHgLCmxzP/zliu9b8EH0LH8Zr/j+cgZew+JEF3g1dzi9Fp1Nw3KpGK0gPuDkcI0aTl/bZCeCODwUB9fb3rtcPhQKNxXgoEBwezcOFC5s6dS3R0NKmpqYSGhvLOO++gUqnIy8ujsLCQxx57jJycHCIjI91VjE739uaDRBh8+eWQGHeH0jEcdjh1uEUivq8pES+G2qMtNlRBcG8IT4K0OzjZ6EuPPn3Pch92IOgCwadrX0q67byv8XX+6SN+3vv85L78lrcIOP+qjh4gwh8wVYGpEk7scD42VJ/9fbUB50jo21jmHwYamRJYiI7Qtc+qXVBJZT1r848y8+p4IgN9qXB3QEIIl2eeecbdIQjRqYYOHUpubi7jx48nPz+f/v37u9bZbDYKCgpYsWIFNpuNrKws5s+fzw033ODaZvr06WRnZ3t1cr6/rI5NReU8PK4/vhofd4dzYUxVLZLv5kR8P1QdALvl9HZ+wRDeD+JHQURf5/Pwvs7EvEXLaVVhIT28vDWry5/3zyPRLy8sJKKt79Fu+//t3Xl8VPW9//HXzGTfCGFfQ4CwJICIGyoginGBukCIIelFBdpeFfTSer0KP4sovYJ4e6tVQGsBW1zBogVv1VYkKBFRURCyKJAQ9j1kX2Yy8/vjhJCEbECSM5l5Px+PeWS2M/P5Ovg9857v+X4PlJ4xwvp5l9M1b5/OMu4ry6+/Fv+w88N7YET9o/aB7dv8DzwiLUH/V7SypRv34mO18O9j+ppdiojUMnnyZNLS0swuQ6TFxMXFkZqaypQpU3C5XDz77LOsX7+e4uJiEhMT8fX1ZdKkSfj7+zNt2jQiIiLMLrnVrUzdh5+PlZ9f09vsUupmL4Xc7Jqj4GevVx8RtfpCRJQRvqNvgY7R54J4cEctPlbJq/t9m4/xb+FCRvEd5ca/s8YCfeFxOJ5pXLcX1f96AeFNG6E/e39AOFjNmgwh0joaDeiNnZJl3bp1rFy5EqvVSnx8PMnJyaxdu5b3338fgLKyMjIyMkhNTeXAgQM88MAD9OnTB4CkpCTGjx/fMi1zQwdOF/O37w7ybyMj6Rymyeci7iY9Pd3sEkRalNVq5ZlnnqlxX/VTps2aNYtZs2bVu/2qVatarDZ3kFtUzt++O8jE4T3oEHLhi401G5fLOPS8+ij42et5B8DlPPfckK5G+I652wjfHStDeHikRiebQP3+BfLxg9AuxqWp7CXnB/jzbp+CvINwZIdxvb659RZr5ah8zTDfscwHSkdAeG/jEtYDbL7N02aRVtZoz93QKVkAFi9ezIcffkhQUBATJkxgwoQJTJo0iUmTJgHGoUPx8fGEhYWRnp7OtGnTmD59esu1yI0t27QXq8XCAzd4/vljRURE2pq3vt5Pqd3J9NY6tVpp/vmj4Kf2GLftxeee5xtsHH7e80q4LKkyiPc3/vo3cflxEbP4BkK7HsalKVwu499/Y4G++LRx6P2Br+lYdALSlp97DYvVCOlnA3vtiwK8uLFGA3pDp2QBGDhwIAUFBfj4+OByubBUO2Rq586d7Nmzh6eeegqAXbt2kZ2dzYYNG4iMjGTu3Lk1VpP1ZIfPlLDm2wNMuao3XbV0u4hb8uR5tSLSMHuFk79u2ceo/h0Z2LUZQ2+FHXJzzp8XfmoPFB479zyL1Rj17tDfmBveof+5EfHQbjokvYWo33dDFgv4BRuX8KZNNcnc9QODu4caiyHWvmR/UbkYoqvae9QK8O161Qzw7XoqwItpGg3oDZ2SBSA6Opr4+HgCAwOJi4sjLCys6rmvvvoqM2fOrLo9bNgwEhISGDJkCMuWLWPJkiU8/vjj9b53Wz+PanVLvjqJy+Xi5h41z0XpDef7Uxs9gze08ZNPPvH4NnrD5+gNbdRpaJrfP3Ye4Vh+GYsmDbvwjV0uKDpRa4G2ylOW5e4Dp+Pcc4M6GHPB+8fVXKAtIspY7Eta1aZNm8wuQZqDrXLNhYh6jn5xlBshva4Av2+z8Vj1qSMWK4R2r38EXgFeWlCjAb2hU7JkZmaSkpLChg0bCAoK4rHHHuOjjz7i9ttvJz8/n6ysLEaOHFm1bfUAHxcXx4IFCxp877Z+HtWzjuaV8s89+0i4sjc3XDW0xmPecL4/tdEzeEMbZ86cyZIlS8wuo0V5w+foDW2U5uVyuVi+OZu+nYK5YUAjI6pOJxz4ig5p70NGXmUo3wtleeeeY/M3DknvHAOD76y2QFs/Y86suI2XX37Z4/t9wZg731CAr7DXH+BzUmHn6iYE+Gqj8GE9ve+0cy4XlBdBaV79l7IGHivNY4BPEKREGj+AtOtR+beXcbRDu57G0UResLZGoy1s6JQsoaGhBAQE4O/vj81mIyIigvx84/QL33zzDdddd12N15oxYwa//e1vGTZsGFu2bCE2NraZm+OeXtm0F6fLxUNjNfdcxJ0tXbpUX9REvNC2nFx+OJjHgruHYLXWcSi5ywUHv4Vdf4P0D6DgCJ3B+BLesT8Mu6favPBo44uktY2dos1Lqd8XwBgNb9/HuNSl3gB/AHK+PD/AY4GwBkbg3THAu1xQVlAZpPMbCNJnat2u9lxXRcPv4RNonObx7CWoI0T0g4Aw8A8j/+h+2tuKjcUw939pvGZ1FqsR0tv1NC5hPYwAXxXoexmn72vjU4IaDeiNnZIlMTGR5ORkfH196d27NxMnTgQgOzubnj171nit+fPns2DBAnx9fenYsWOjI+ie4Hh+KW9/vZ9JI3rQKyLI7HJERESklhWp2bQL9CV+RLVFrFwuOLIddq2FtA8gbz/Y/IxD04dM4kdXHwYOu9K8okWk9TQpwB+uZwR+C+xc07QAf3YufLteFx7gnU4oL6gnVNcO3LVDdmUor1FjHXyDqwXsMAjpAh0H1Azd1S/+7Wo+v5FpPEczMmhf/Qi4sgLIO2Ss8J93wPiRJO+gcTn0HWSsh4ryWjUGVQvvlSPwNUbjuxsLF7qxRgN6Y6dkSUpKIikp6bztfvGLX5x3X2xsLO+8887F1Nlm/enzLBxOFzNv7G92KSIiIlLLgdPFfLzrKL8a048gXxsc3QVpa41gnpsNVh/odxPcOBcGjTe+aAJOD1/nQEQugM0X2kcal7rUFeDzDhh/92+Bne/VGn22GCPF1cJ7x9O5sM+v1gj3mZohvPpCeHXxC6kZoMO6Q+fBdQTrsFr3hRsBu7Xn3fuHQudBxqUuTqex/kf+wXPBPe+Q8d827yAcT6+5GOdZQR3PjcLXNRof0gWs1pZtWwM8/yB+E50sLOONrTncPbwHkR2CzS5HRBqxZs0as0sQkVb2ly/30d9ymJmWbbBkPZz8CSw2iBoDo38Dg36meeMeTP2+tIpGA7wDCuoZgT/wFez6G51cFecH53a9oMuQekJ1HaHb0+ZvW60Q2sW49Lii7uc4yipH3quNvp8N9Kf2QFYKlBfWel0f48eL6vPfa18qf6xtCR72KbmX177IotzhZOaNmnsuIiLiVk7tpWzHe9zzzRs86bcfvrJAn1FwzQMQcxcEdzS7QhHxFjafc6PldalwkJGZyeDYIa1blyfw8YeIvsalLi6XcQRC3sHKIH+g5mj8/q8qV/mvNb/eL7RaYK9+OH3laHxYj4teZ0ABvYWcLipn1ZYc7rysO307ece53kXauoSEBFyuRg4PE5G2KzcH0t43DmE/sgN/IM81gIPXzqfn9UkQ2tXsCqWVqd+XNsHmo4UnW4rFAoHhxqVrPT+AOCuMQ+XPzoWvPRp/+DsoPlX7hY1D5WuE914w8oFGS1JAbyF//iKLEnsFs27S3HMRERHT5B82FnlLWwsHvzHu6z4CZ9wC7tncFWdoD9bedr25NYqIiPuy2oxD3sO6Q6+r635OebGxvzk7Al99NP5YOvz0T3CUKKCb5UxxOX/5ch8Thnajf+dQs8sRERHxLoXHIf3vxkJv+7cALug6FMY9BbETISKKT9OO8m3uNpbcXs9hjyIiIk3lF2ScarNjPYOzZ09j1wQK6C1gxeZsisorePimaLNLEZEL8NBDD5ldgohcrKJTkLHOGCnft9k4XVCnwcbq67GTzvvStCI1mx7hgdwa28WkgsUdqN8XkVZhsRgr4TeBAnozyyuxszJ1H+OHdmVgV42ei7Qls2bNMrsEEbkQJWcg80NjpDwrxVjEp0N/GP2fMGSScfqgOqQdzuOrrNPMHT8IH5t5p9IR86nfFxF3o4DezF5P3UdBmYNZN2r0XKStueGGGzh+/LjZZYhIQ8oK4MePjFC+dwNUlEN4JFz/iDFS3nWoMVLRgBWb9xHkZyPxqnpWTBavoX5fRNyNAnozKii1s3xzFrfEdCGme9MOYRAR93HixAmzSxCRupQXwU+fGIev7/4XOEohrCdc/StjpLz7iEZD+VnHC0pZv+MwSVf3ol2gbwsXLu5O/b6IuBsF9Gb01y055Jc6eGScRs9FREQuib0U9vzLGCn/6WOwFxunrBlxHwyJh55XgfXCD09/Y0sOdqeT+6+PaoGiRURELo0CejMpLHPw2hdZjBvUmSE92pldjohchJiYGLNLEPFujnLY+5kxUp75DygvgKAOcNkU4/D1yOsu6VzApfYK3ti6n3GDOhPVMbgZC5e2Sv2+iLgbBfRmsmpLDmeK7Tys0XORNuu9994zuwQR71Nhh+xNsOt9yFwPpXkQEA6xdxuHr/cZA7bm+bry9+2HOF1UzvRRGj0Xg/p9EXE3CujNoLjcGD2/YUAnhvcKN7scEblITz31FKtXrza7DBHP56yAnFTj8PX0v0PJafAPg0ETjJHyvmPBx69Z39LlcrFi8z4GdQ3l2r4dmvW1pe1Svy8i7kYBvRm8+dV+TheVa+65SBu3Zs0as0sQ8VxOJxzYahy+nv53KDwGvsEw8HZjpLzfOPANaLG3T91zih+PFfD85GFYmrignHg+9fsi4m4U0C9RSXkFr36exejojlwR2d7sckRERNyHywWHtlWOlH8A+YfAJwCibzFCefSt4BfUKqWsSM2mY4gfd1zWvVXeT0RE5GIooF+it7/ez8nCMh4ZN8LsUkRERNyCxVEK/5oHae/Dmf1g84P+N8PNT8PA28A/tFXr2XuikM8yjzP75mgCfC9+kTkREZGWpoB+CUrtFbyyaS/X9u3AVX0izC5HRC6GoxzyDsDpLL5dNQ/SPjBG+Hz8a/71Dai8Xe2+S1hNWi6QywUV5cb5rx1n/5bV+lsKFeUEHz0GHYCw7sa8Zh3O3Or887NhyxLoeyOMnQMDx0OgeWu0rEzNxs9m5d9GRppWg7inlJQUs0sQEalBAf0SvPvNAY4XlPHilMvNLkVEGlJeDLnZcDobTmdVXs8ybucdAJcTgMM/2rlij2/TX9fqU3eY9/EHn8B67m/gr+8FbGPzv6hzQF+0CgdUlNUMwzUCclndgdlRVmu7xrap/FvXNk3UG2BT5Q2/ECOoh3WHsB7G39Bu566H9YCgCIX4ZlYe0gP+c7fx39ZkZ4rL+du2Q9w1vDsdQ/zNLkfcTFpaGjfccIPZZYiIVFFAv0hljgqWpezl6j4RjOxr/hcQEa9XkmsE7urh+2wgLzxa87mB7aF9FPS8CobdAxF9oX0Ud/a5DtfRb+oOjPa6Qmk9o7hnr5cXQvHJ+p97qWwX+gNAAF1zz8BPQQ3UX8/otKvi0us9+8NCffUFhDXyo4RfrftrP8e47NuTSZ/2vlBwBPIPG/Oe8w9DVopxX+UPMjX+O1YF+G41w/zZ68GddMTEBXD6hblFOAd4++sDlNgrdGo1qdPMmTN56KGHzC5DRKSKAvpFWvPtQY7ml/I/CZdpNViR1uByQeHxWgG82mh4SW7N54d0NYJ3/3EQEWUE8oi+xvXABhZ07BLbsu04q8Yh22VgL2la6K9xXyM/GJSeOe8HhlCHHfyD6wjBAca5p30aCcENBWyfgDq2rxauW6mvLMkLgMGD636wwgFFxyuD++GaAT7/MBz42gjxFeU1t7P6GP+mao/GVw/2od3AdgFHYEiLs1c4+euWfVzXrwODu4WZXY6IiEijFNAvQrnDybKUvYzoHc71/XUuVZFm46wwwtJ5AbzyYi8691yLFdr1NEJ3zN2V4bsygLfvA37BpjWjSSyWc6O+rWh3RgaD6wuv3sDmcy5Y18flguJT1YL7oZph/lga7P4n2ItrbWiBkM7nB/jQ7tXCfHdjKoNJnE4n8+fP58cff8TPz4/f/e53REaem5f9wQcfsHz5ckJDQ5k4cSIJCQnY7Xbmzp3LoUOHKC8v58EHH2TcuHGmteFCfLTrKEfySvnd3UPMLkVERKRJFNAvwtrvDnLoTAnPThqq0XORC+UohzM5dc8HP5NTc+TS5meE7fZR0Ge0Eb4rD0cnvLcxWtuM5s+f36yvJ22UxQLBHY1Lt8vqfo7LBaV554/CF1TePp0F+74wnlNbYES1AN+tjtH47i22yvmnn35KeXk57777Ltu3b2fRokUsW7YMgNOnT/Piiy/y/vvvExYWxv3338+1117L1q1bCQ8P5/nnnyc3N5eJEye2mYC+YnM2UR2DuXFgZ7NLETelfl9E3E2jAb2xX9vXrVvHypUrsVqtxMfHk5ycDMDdd99NaKjxBaNnz54sXLiQnJwcnnjiCSwWC9HR0Tz11FNYW3ORo2Zgr3CyJGUPl/UKZ0x0R7PLkaZwVtSaQ1xa63ZJrUOMG7pde1vjElVSDJvDwTfIuPgFgW+wMVJ23vWzz6m8r8b1YOM5PgFte9Gq8qK6A/jpbMg/WHMOsF+IEbg7D4ZB488F8Ii+RlBpxXm/99xzT6u9l7RxFouxKnlgOHSJqf95ZYWVc+EPQf6R80fjD20z1imozT+scjG7ug6n7w5dL25EeNu2bYwePRqA4cOHs2vXrqrHDh48yKBBgwgPN1ZbHzp0KDt27OC2227j1ltvrXqezdY25uJvy8ll+4EzPHNXLFZrG+5PpUWp3xcRd9NoQG/o13aAxYsX8+GHHxIUFMSECROYMGECAQEBAKxatarGay1cuJDZs2dzzTXXMG/ePDZs2EBcXFwzN6llffD9IQ6cLuHpO2M1en6hnBWNhOBawbdJgbms3tBcddtpv4SiLdVW1q78W/22XwgEd6LcVkRAgI9RU+FRY9Vwe4lxSHZ5sbEi9QW/b1AdAT+42v11Bfw6wv7Z67W3v9R/v8Wna66MXj2QFx6r+dzACCNw974GIpJqzgcP7uQ2P0bExMTgcrnMLkM8iX8I+EdDx+j6n2MvNUJ87UXtzv7dm2n8P1X9h635dYzMN0FhYSEhISFVt202Gw6HAx8fHyIjI9mzZw8nT54kODiYLVu20KdPH4KDg6u2feSRR5g9e3aj7+N0OsnIyLioGpvLCynHCPGzMiS4qEVqKS0tNb2NLc0b2hgTE0N6errZZbQob/gc1UbP4A1tbMo0w0YDekO/tgMMHDiQgoICfHx8cLlcWCwWMjMzKSkpYfr06TgcDn7zm98wfPhw0tLSuPrqqwEYM2YMqampbSqgOyqcLNm4hyE9wnS4XH0cZcb8zCM7Ki/bGXDiJyOgOh2X8MJng3JAHeekDjAOBw3uVP/jF3U70FjwqQnh8VBGBmEN/Q/nrDDmq5YXG3/Pu15UGehrXa96XuV95cXGDwBnr5+9/2JWBPetPpofVHeor3a904mjkFZwLoyXnqn5eqHdjcAdHVczgLePMvX8xyJuzzegcvpGA6uMVziMkH42uF+kkJAQiorOreXgdDrx8TG+CrRr1445c+bw8MMP07VrV2JjY2nf3lhQ8ciRI8ycOZPk5GTuuOOORt/HarWautbBwdxiUvdn8cvRfRkxrGXqyPCC9Ry8oY3QtC/MbZk3fI5qo2fwhjY2RaMBvaFf2wGio6OJj48nMDCQuLg4wsLCCAgIYMaMGSQkJLBv3z5++ctf8vHHH1cFeIDg4GAKCgoafG93+AW+ug17C9h3qph5N3YhMzOzWV6zLf9SZHGU4n9mD4G5mQTk/khA7o/45+3FUnk6pgrfUErbD6So561YA0Jx2fxx2vxw2fxxWf1w2vxx+fjjtPob99kq76v212ULwGn1M1ZQbu5R1orKS9XgthMorrxcmAv/HK1ASOUF4/9EH+Bi145yVmCtKMVSUYrVYVwsFSXG9YpSLI5z162OEiz1XLcWF2LJP2lcryg991hFGR0sVsqDulEe0gN7j5soD+1JeYhxsQd3x+UTcH5deUDeEeDIRTas9bXV/x+bqi33OU3l2W0MAetALvbry4gRI9i4cSPjx49n+/btDBgwoOoxh8PBjh07ePPNN3E4HEybNo1f//rXnDx5kunTpzNv3jyuvfba5mlGC/vrlhwsFgv3XtfH7FJEREQuSKMBvaFf2zMzM0lJSWHDhg0EBQXx2GOP8dFHHzFu3DgiIyOxWCxERUURHh7OiRMnasw3LyoqIiys4VOemP0LfHUVThcz/28Tg7uFMe2WK5rt8PY280tRWSEc2wWHt1eNjHPix3PnRg6MgO7DYcgE6DYcul2GrX0fgi0W9reVNl6CNvM5Xiynk8yMNAbHDqV5l2VzL2PHjvXszxEv+LeKd7TxYsXFxZGamsqUKVNwuVw8++yzrF+/nuLiYhITE/H19WXSpEn4+/szbdo0IiIi+N3vfkd+fj5Lly5l6dKlALz22mtV09ncTVGZg7e/3s9tQ7rSI9y8FfOlbRg7dqzZJYiI1NBoQG/o1/bQ0FACAgLw9/fHZrMRERFBfn4+7733Hj/99BPz58/n2LFjFBYW0qlTJ2JiYti6dSvXXHMNn3/+OSNHjmzRxjWnD384TNbJIpb9fITnzz0vzYOjO2uG8ZO7gcq5ucGdjTA+6FwYp11Pt5lHLC3AajWOYvBwZ8OHiKeyWq0888wzNe7r169f1fVZs2Yxa9asGo8/+eSTPPnkk61SX3N4b9tBCkodzBjVwJQBkUrq90XE3TT6jbuxX9sTExNJTk7G19eX3r17M3HiRADmzJlDUlISFouFZ599Fh8fHx5//HF++9vf8r//+7/07du3xqqw7szpdPHSZ3sY2CWUW2O7ml1O8yo+DUd/qBnGT2edezy0e+XIeLwRxLsNh9CuCuPikR566CE2btxodhkicpGcThcrU7MZ3iucEb3bm12OtAHq90XE3TQa0Bv7tT0pKYmkpKTztvv9739/3n1RUVG88cYbF1OnqT7adZQ9xwt5Ofnytn2qlqKTRgCvCuM7jPNOn9WuN3S/DIYnnxsZD9FieOI9UlJSzC5BRC7BZ5nH2XeqmJduGWh2KdJGqN8XEXfj+cesXiJj9Hw3/TuHcPuQbmaX03QFR40AXj2M5x8893j7KOgxAq6cdm5kPCjCvHpFREQu0fLN2XRrF8BtQzzsaDcREfEaCuiN+Gf6MTKPFvDilOHY3HH03OUyTrlzNoSfDeSFRyufYIEO/aH3SONQ9W6XQddhOu2ViIh4lPTD+WzJOsUTtw/C12ZtfAMRERE3pIDeAJfLxR837KZvx2B+Nqy72eUYYfzMfuMw9ephvPik8bjFCh0HQt+x1cL4UOMc4SLSqPT0dLNLEJGLtCI1m0BfG0lX9Ta7FGlD1O+LiLtRQG/ApxnHST+Sz+8TLmv90XOnE3Kzzw/jpWeMx60+0GkwDLjtXBjvMgT8glq3ThEPsnr1ap566imzyxCRC3SioIx12w+TeFUv2gX5ml2OtCHq90XE3Sig1+Ps6HlkhyDuGt7Co+fOCji1t2YYP/oDlOUbj9v8oHMMxNx1Lox3jgVf9zwHrUhbNX/+fH1RE2mD3vgqh/IKJ9Ou72N2KdLGqN8XEXejgF6PlB9PsPNQHosnD8OnJeaylebBjneJ/PZNWLsb7EXG/T4Bxkj40IRzYbzTYPDxa/4aRERE2rhSewVvbs3hpkGd6dspxOxyRERELokCeh1cLhcvbthNz/aBTLy8R/O++NGd8M1y+GE12Iuwhg+Ay//NCOLdhxtzyG36WERERJpi3Y7DnCwsZ8aoKLNLERERuWRKgnX4YvdJth84w8JJQ5tnJVhHGaSvg2/+DAe+MkbJh0yGq6aTnR/E4MGDL/09ROSSLVmyxOwSROQCuFwuVmzOZlDXUK7r18HscqQNUr8vIu5GAb2Ws6Pn3dsFED+i56W92Jn98O1K+O6vxkrrEX3hlv+G4cnnzjmen3HpRYtIs4iNjTW7BBG5AFv2niLzaAGL44dhsbjhqVDF7anfFxF3o4Bey5d7T7EtJ5cFdw/Bz+ciRs+dTti7wTiM/aePwWKBAbfDVTOg741g1blZRdzV2LFjcblcZpchIk20fHM2HYL9uLOlF3MVj6V+X0TcjQJ6LS9u2E3XsADuufICR8+LTsH2N+DbFZC7D4I7w5j/hBH3QXivFqlVRETEW2WfLGJD5nEeGRdNgK/N7HJERESahQJ6NV9lneLr7NPMvyMGf58m7OxdLji0zZhbvmstVJRB5PUwbh4MukMrr4uIiLSQlanZ+Nms/NvI3maXIiIi0mwU0Kv544bddAr1Z8rVjezsy4th13tGMD+yA/xCYMRUuHIGdIlpnWJFpNklJCSYXYKINEFesZ013x7kjss+WSU3AAAc40lEQVS60zk0wOxypA1Tvy8i7kYBvdI3+07z5d5T/PZnMfUfKndyt3EI+/Y3jfOYd46BCb+HYYngH9q6BYtIs3v66afNLkFEmuCdb/ZTYq9g+qg+ZpcibZz6fRFxNwrolf64YTcdQ/xIrj16XuGAH/8B3y6HrBSw+kLMnXDVL6D3tcYicCLiESZPnkxaWprZZYhIAxwVTv7y5T5G9o0gtns7s8uRNk79voi4GwV04Lv9uXyx+yRzxw8i0K9y9LzgKGz7C2x7HQoOQ1hPuOlJuPxeCO1iar0i0jLS09PNLkFEGvFx2lEO55Xy9F1DzC5FPID6fRFxNwrowEsbdhMR7MfPr+4N2V8Yc8szPwSnA/qNgwn/A9G3gk3/uURERMy0fHM2kR2CuGlQZ7NLERERaXZenzh3HDjDtz/msGTITwT/+f/ByR8hIByueQCunA4d+pldooi0kk6dOpldgog04Lv9uXy//wzz74jBZtUUM7l06vdFxN14d0A/upPc1YvYGvAvgvaUQfcRcNdSGDIJfAPNrk5EWtmmTZvMLkFEGrBiczahAT4kXNnL7FLEQ6jfFxF3YzW7gFbnKIMf1sDyW+GVUYzM/4R9XW6BX34Gv9oIl/9c4VzES7388stmlyAi9Th8poSPdh1lylW9CPb37vEFaT7q90XE3XhPQM/NgU/nw//GwNpfQNFx1nR4gJstr9Jz2krocYXZFYqIyZYuXWp2CSJSj79s2YfL5eK+6/qYXYp4EPX7IuJuPPsnaKcT9m4wFn376RPjlGgDx8NVM8gIHMFjf0xl9s3RhAX4ml2piIiI1KO43MHbW/dz25Cu9GwfZHY5IiIiLabRgO50Opk/fz4//vgjfn5+/O53vyMyMrLq8XXr1rFy5UqsVivx8fEkJydjt9uZO3cuhw4dory8nAcffJBx48aRlpbGAw88QJ8+fQBISkpi/Pjxzd+qolOw/Q34dgXk7oPgzjDmP+GK+6FdTwBefvM7Qv19mHZdVPO/v4iIiDSbv207SH6pgxmjtM8WERHP1mhA//TTTykvL+fdd99l+/btLFq0iGXLllU9vnjxYj788EOCgoKYMGECEyZM4NNPPyU8PJznn3+e3NxcJk6cyLhx40hPT2fatGlMnz69+VvicsGhbcZo+a61UFEGkdfDuHkw6A7w8at66k/HCvjHriPMurE/7YI0ei4ihjVr1phdgojU4nS6WJG6j8t6tmNE7/ZmlyMeRv2+iLibRgP6tm3bGD16NADDhw9n165dNR4fOHAgBQUF+Pj44HK5sFgs3Hbbbdx6661Vz7HZbADs2rWL7OxsNmzYQGRkJHPnziUkJKTe93Y6nWRkZDRYn8VRStj+f9J+z98IzP2RCp8g8qJ+xpl+kygLrzxF2u69NbZZtOkYATYLozrZG339llZaWmp6DS1NbfQM3tDGsrIyj2+jN3yO3tDGwYMHm11Cq0n56TjZJ4t4ccpwLBadWk1ERDxbowG9sLCwRoi22Ww4HA58fIxNo6OjiY+PJzAwkLi4OMLCwmps+8gjjzB79mwAhg0bRkJCAkOGDGHZsmUsWbKExx9/vN73tlqt9X8JObkbvlkO29+CsjzoHAMTfo9tWCIR/qFE1POae44X8nlOFg/c0I9rLh/UWPNbXEZGhsd/0VIbPYM3tDEmJgaXy2V2GS3KGz5Hb2jjxWps2toHH3zA8uXLCQ0NZeLEiSQkJDS6TUtbvjmbrmEBjB/ardXeU7xHQkKCx/f7ItK2NLqKe0hICEVFRVW3nU5nVTjPzMwkJSWFDRs28Nlnn3H69Gk++ugjAI4cOcK9997LXXfdxR133AFAXFwcQ4YMqbqenp5+YdVWOCB9HfzlTnj5SuNw9ug4mPYxPPglXPUL8A9t8CWWbNxDgI+NX2gem4iIeJnq09YeffRRFi1aVPXY6dOnefHFF1m1ahVvvPEG69ev5+DBgw1u09Iyj+aTuucU914Xia/Ne048IyIi3qvRvd2IESP4/PPPAdi+fTsDBgyoeiw0NJSAgAD8/f2x2WxERESQn5/PyZMnmT59Oo899hiTJ0+uev6MGTP44YcfANiyZQuxsbFNq7LgKKQ8By8MhdVT4dReuOm38Jt0mLwcIq81VmhvRPbJIv6+/RBTr42kQ4h/095bRETEQzQ0be3gwYMMGjSI8PBwrFYrQ4cOZceOHY1OdWtJKzZnE+BrJfnq3q32niIiImZq9BD3uLg4UlNTmTJlCi6Xi2effZb169dTXFxMYmIiiYmJJCcn4+vrS+/evZk4cSKLFy8mPz+fpUuXVp1f8rXXXmP+/PksWLAAX19fOnbsyIIFCxp8b4u9GFbfB5kfgtMB/cbBhN9D9C1gu/AzxC3ZuAc/Hyu/HN33grcVEc/30EMPmV2CSItqaNpaZGQke/bs4eTJkwQHB7Nlyxb69OnT6FS3ujRlDZnGnCmp4P3vDhHXP4QjOXs5ckmv1vy8Ya0Db2jjr371K49vozd8jmqjZ/CGNjZlCl6jKddqtfLMM8/UuK9fv35V15OSkkhKSqrx+JNPPsmTTz553mvFxsbyzjvvNFrUWf4FOZCVAtc8AFdOhw79Gt2mPvtPFfP+94e4/7o+dArV6LmInG/WrFlmlyDSohqattauXTvmzJnDww8/TNeuXYmNjaV9+/YNblOfBteQaaIXP92N3eniNz8bQf/O9S8oaxZvWOvAG9o4e/Zsj2+jN3yOaqNn8IY2NoVbT+gqC42ERzPh1v++pHAOsDRlDzarhX8fo9FzEanbDTfcYHYJIi2qoWlrDoeDHTt28Oabb/Lcc8+RlZXFiBEjGtympZQ5Klj1VQ5jB3Zyy3AunkP9voi4mws/TrwVuXyDwDfwkl/nwOli3tt2kH8bGUnnsIBmqExEPNGJEyfMLkGkRTU2bc3X15dJkybh7+/PtGnTiIiIqHOblrZ+xxFOFpYxQwu6SgtTvy8i7satA3pzWbZpL1aLhX+/QaPnIiLivRqbtjZr1qzzpnrUtU1LcrlcLN+czYAuIYzq37HV3ldERMQduPUh7s3h8JkS1nx7gHuu6km3dpc+Gi8inismJsbsEkS83ldZp8k4ks/066OwNOEMLSKXQv2+iLgbjw/or2zaC8CDY/ubXImIuLv33nvP7BJEvN7yzdlEBPtx9+U9zC5FvID6fRFxNx4d0I/mlfLO1weYfEUveoRr9FxEGvbUU0+ZXYKIV9t3sogNmcf4+TW9CfC1mV2OeAH1+yLibjw6oL/6+V6cLhcPjb20FeBFxDusWbPG7BJEvNrrX+7Dx2ph6shIs0sRL6F+X0TcjccG9OMFpby1dT+TRvSgV0SQ2eWIiIhIA/JK7Kz+9gB3DOuuM66IiIjX8tiA/qdNWTicLmbeqLnnIiIi7m71NwcoLq9guk6tJiIiXswjA/rJwjLe2JrDXcO7E9kh2OxyRKSNSElJMbsEEa/kqHDy+pf7uDoqgiE92pldjngR9fsi4m48MqC/9kUW5Q6nRs9F5IKkpaWZXYKIV/pn+jEOnSlhhkbPpZWp3xcRd+NxAf10UTmrtuRwx2Xd6dcpxOxyRKQNmTlzptkliHil5Zuz6R0RxM2Du5hdingZ9fsi4m48LqAv35xFib2Ch2/S6LmIiIi7237gDNtycrn/uj7YrBazyxERETGVRwX0M8Xl/OXLHCYM7Ub/zqFmlyMiIiKNWLE5m1B/H+65qpfZpYiIiJjOowL6itR9FJY5ePimaLNLEZE2aP78+WaXIOJVjuSV8I+dR7jnql6E+PuYXY54IfX7IuJuPCag55XYWZmaze1DujKwq0bPReTC3XPPPWaXIOJV/rolB6fLxf3X9TG7FPFS6vdFxN14TEB/PXUfBaUOZmnuuYhcpJiYGLNLEPEaxeUO3tq6n1tiutIrIsjscsRLqd8XEXfjEQG9oNTO8s1ZxMV0Iba7zp8qIiLi7tZ+d4i8EjszRuvUaiIiImd5RED/65Yc8ksdPKK55yIiIm7P6XSxIjWboT3acWVke7PLERERcRttPqAXljl47YssbhrUmaE9NXouIhdv7NixZpcg4hU27T5B1okiZoyKwmLRqdXEPOr3RcTdtPmA/sZXOZwptvPIOI2ei8ilWbp0qdkliHiFFZuz6Rzqz/ih3cwuRbyc+n0RcTeNBnSn08m8efNITExk6tSp5OTk1Hh83bp1TJw4kfj4eN56660Gt8nJySEpKYnk5GSeeuopnE7nJRVfXO7gtc+zuGFAJ4b3Cr+k1xIReeihh8wuQcTj/XSsgC92n+S+6/rg59PmxwmkjVO/LyLuptE946effkp5eTnvvvsujz76KIsWLarx+OLFi1m5ciVvv/02K1euJC8vr95tFi5cyOzZs3nrrbdwuVxs2LDhkop/a+t+ThWVa/RcRJpFSkqK2SWIeLwVm7Px97GSfHVvs0sRUb8vIm6n0YC+bds2Ro8eDcDw4cPZtWtXjccHDhxIQUEB5eXluFwuLBZLvdukpaVx9dVXAzBmzBi+/PLLiy68pLyCVzZlMap/R67QAjMiIiJu71RhGWu/P8SkET1pH+xndjkiIiJux6exJxQWFhISElJ122az4XA48PExNo2OjiY+Pp7AwEDi4uIICwurd5uzAR4gODiYgoKCBt/b6XSSkZFR52MfpOdxsrCMu/pH1PuctqC0tLRN198UaqNn8IY2Ah7fRm/4HL2hjYMHDza7hIvy1tb9lDucTL++j9mliIiIuKVGA3pISAhFRUVVt51OZ1U4z8zMJCUlhQ0bNhAUFMRjjz3GRx99VO82Vuu5AfuioiLCwsIafG+r1Vrnl5BSewXvr93IyL4RJIwd0Xgr3VhGRkab/aLVVGqjZ/CGNqanp3t8G73hc/SGNrZFZY4K/vpVDmMGdCK6S6jZ5YgARr8vIuJOGj3EfcSIEXz++ecAbN++nQEDBlQ9FhoaSkBAAP7+/thsNiIiIsjPz693m5iYGLZu3QrA559/zpVXXnlRRa/+9gDHC8o091xEmtXq1avNLkHEY/3fD0c4UVDGjFFRZpciUkX9voi4m0YDelxcHH5+fkyZMoWFCxcyZ84c1q9fz7vvvkuPHj1ITEwkOTmZpKQkCgoKmDhxYp3bADz++OO89NJLJCYmYrfbufXWWy+44DJHBctS9nJ1nwiu7dvhwlssIlKP+fPnm12CiEdyuVws35xN/84hjInuaHY5IlXU74uIu2n0EHer1cozzzxT475+/fpVXU9KSiIpKem87WpvAxAVFcUbb7xxMXVWeW/bQY7klfL85Muq5rOLiIiI+/o6+zRph/N5duJQ7btFREQa0KZOQFrucLJ0415G9A7n+v4aPRcREWkLlm/OJjzIl4mX9zC7FBEREbfWpgL62u8OcuhMCY+Mi9Yv8CLS7JYsWWJ2CSItyul0Mm/ePBITE5k6dSo5OTk1Hl+3bh0TJ04kPj6et956CwC73c6jjz7KlClTSE5OZu/evRf0nvtPFfOvjGP8/JreBPrZmq0tIs1B/b6IuJs2E9DtFU6WpOzhsp7tuGFAJ7PLEREPFBsba3YJIi3q008/pby8nHfffZdHH32URYsW1Xh88eLFrFy5krfffpuVK1eSl5fHpk2bcDgcvPPOO8ycOZMXXnjhgt5z5ZfZ2CwW7r22TzO2RKR5qN8XEXfTZgL6B98f4sBpjZ6LSMsZO3as2SWItKht27YxevRoAIYPH86uXbtqPD5w4EAKCgooLy/H5XJhsViIioqioqICp9NJYWFh1alWmyK/1M7qbw7ws2Hd6BIW0KxtEWkO6vdFxN00fS9rIkeFkyUb9zCkRxg3DepsdjkiIiJtUmFhISEhIVW3bTYbDoejKnRHR0cTHx9PYGAgcXFxhIWFUVRUxKFDh7j99tvJzc3llVdeafR9nE4nGRkZvJ92hqLyCm7qaSEjI6PF2mWG0tJSj2tTbd7QRsDj2+gNn6Pa6Bm8oY2DBw9u9DltIqCv/+Ew+04V86epV2j0XERE5CKFhIRQVFRUddvpdFaF88zMTFJSUtiwYQNBQUE89thjfPTRR2zfvp1Ro0bx6KOPcuTIEe677z7Wr1+Pv79/ve9jtVoZMHAQ/1i3kav6tOfOUcNbvG2tLSMjo0lftNoyb2gjNO0Lc1vmDZ+j2ugZvKGNTeH2h7hXOF289NkeBncLIy6mi9nliIgHS0hIMLsEkRY1YsQIPv/8cwC2b9/OgAEDqh4LDQ0lICAAf39/bDYbERER5OfnExYWRmhoKADt2rXD4XBQUVHR6Hv9K/0oB3NLmDEqqmUaI9IM1O+LiLtx+xH0/9t5hKwTRSz7+QiNnotIi3r66afNLkGkRcXFxZGamsqUKVNwuVw8++yzrF+/nuLiYhITE0lMTCQ5ORlfX1969+7NxIkTsdvtzJ07l+TkZOx2O7/+9a8JCgpq9L2Wb86mZ/tA4mK6tkLLRC6O+n0RcTduHdBdLhcvbdjNgC4h3BqrHbyItKzJkyeTlpZmdhkiLcZqtfLMM8/UuK9fv35V15OSkkhKSqrxuJ+fHy+++OIFvU+p3ck3+3J5csJgbFb9uC7uS/2+iLgbtz7EvbDcye7jhTx8UzRW7eBFpIWlp6ebXYKIRzhTWkGIvw+JV/UyuxSRBqnfFxF349YB/XRJBf06BTN+aDezSxEREZEmKihzknBlT0IDfM0uRUREpE1x64Be5nDx8E3ROjxORFpFp06dzC5BxGNMu06Lw4n7U78vIu7Greeg92zny3XDu5tdhoh4iU2bNpldgohH6NXOl94dGl9ITsRs6vdFxN249Qh6kK9VK7eLSKt5+eWXzS5BxCME+Lr11wuRKur3RcTdaA8qIlJp6dKlZpcgIiKtSP2+iLgbBXQRERERERERN6CALiIiIiIiIuIGFNBFRCqtWbPG7BJERKQVqd8XEXejgC4iIiIiIiLiBhTQRUQqJSQkmF2CiIi0IvX7IuJuFNBFRERERERE3IACuoiIiIiIiIgbsLhcLpfZRdRn+/bt+Pv7m12GiIiIW/Hx8SE6OtrsMuql/beIiMj5mrL/duuALiIiIiIiIuItdIi7iIiIiIiIiBtQQBcRERERERFxAwroIiIiIiIiIm5AAV1ERERERETEDSigi4iIiIiIiLgBBXQRERERERERN+BjdgEN2bFjB//zP//DqlWrzC6l2dntdubOncuhQ4coLy/nwQcfZNy4cWaX1awqKip48sknyc7OxmazsXDhQnr37m12WS3i1KlTTJo0iRUrVtCvXz+zy2l2d999N6GhoQD07NmThQsXmlxR83v11Vf57LPPsNvtJCUlkZCQYHZJzWrt2rW8//77AJSVlZGRkUFqaiphYWEmV9Z87HY7TzzxBIcOHcJqtbJgwQKP+/+xvLycOXPmcODAAUJCQpg3bx59+vQxu6zzaP/dtnnL/tvT992g/bcn8PT9t/bd53PbgP7aa6+xbt06AgMDzS6lRaxbt47w8HCef/55cnNzmThxosft4Ddu3AjAO++8w9atW1m4cCHLli0zuarmZ7fbmTdvHgEBAWaX0iLKysoAPPKL9llbt27l+++/5+2336akpIQVK1aYXVKzmzRpEpMmTQLg6aefJj4+3mN27mdt2rQJh8PBO++8Q2pqKi+88AIvvfSS2WU1q9WrVxMUFMTq1avJyspiwYIFLF++3OyyatD+u+3zhv23p++7QftvT+Hp+2/tu8/ntoe49+7d2+M+nOpuu+02/uM//qPqts1mM7GalnHzzTezYMECAA4fPkzHjh1NrqhlPPfcc0yZMoXOnTubXUqLyMzMpKSkhOnTp3Pvvfeyfft2s0tqdps3b2bAgAHMnDmTBx54gLFjx5pdUovZuXMne/bsITEx0exSml1UVBQVFRU4nU4KCwvx8XHb36Av2p49exgzZgwAffv2Ze/evSZXdD7tv9s+b9h/e/q+G7T/9jSeuv/Wvvt8bvtf4NZbb+XgwYNml9FigoODASgsLOSRRx5h9uzZJlfUMnx8fHj88cf517/+xR//+Eezy2l2a9euJSIigtGjR/OnP/3J7HJaREBAADNmzCAhIYF9+/bxy1/+ko8//tijOtDc3FwOHz7MK6+8wsGDB3nwwQf5+OOPsVgsZpfW7F599VVmzpxpdhktIigoiEOHDnH77beTm5vLK6+8YnZJzW7w4MFs3LiRm2++mR07dnDs2DEqKircKiRq/+0ZPHn/7Q37btD+29N46v5b++7zue0Iujc4cuQI9957L3fddRd33HGH2eW0mOeee45PPvmE3/72txQXF5tdTrP629/+xpdffsnUqVPJyMjg8ccf58SJE2aX1ayioqK48847sVgsREVFER4e7nFtDA8PZ9SoUfj5+dG3b1/8/f05ffq02WU1u/z8fLKyshg5cqTZpbSI119/nVGjRvHJJ5/w97//nSeeeKLqEE9PER8fT0hICPfeey8bN24kNjbWrcK5t9D+u23zhn03aP/tSTx5/6199/kU0E1y8uRJpk+fzmOPPcbkyZPNLqdFfPDBB7z66qsABAYGYrFYPO6L5Jtvvskbb7zBqlWrGDx4MM899xydOnUyu6xm9d5777Fo0SIAjh07RmFhoce18YorruCLL77A5XJx7NgxSkpKCA8PN7usZvfNN99w3XXXmV1GiwkLC6taDKldu3Y4HA4qKipMrqp57dy5kyuuuIJVq1Zx880306tXL7NL8jraf7d93rDvBu2/PYkn77+17z6f5xzj0sa88sor5Ofns3TpUpYuXQoYC+t40mIlt9xyC3PmzOHnP/85DoeDuXPn4u/vb3ZZcoEmT57MnDlzSEpKwmKx8Oyzz3rU4XEAN954I9988w2TJ0/G5XIxb948j/oyelZ2djY9e/Y0u4wWc//99zN37lySk5Ox2+38+te/JigoyOyymlVkZCQvvvgiK1asIDQ0lP/+7/82uySvo/23tBXaf3sOT95/a999PovL5XK1Um0iIiIiIiIiUg8d4i4iIiIiIiLiBhTQRURERERERNyAArqIiIiIiIiIG1BAFxEREREREXEDCugiIiIiIiIibsCzzrUgIk2ydetWZs+eTf/+/avua9++PX/84x8v6XWfeOIJxo8fz5gxYy61RBEREalF+28Rz6eALuKlRo4cyR/+8AezyxAREZELoP23iGdTQBeRKlOnTiUqKors7GxcLhd/+MMf6NSpE4sWLWLbtm0A/OxnP+O+++5j3759PPnkk9jtdgICAqq+LLz77rv8+c9/prCwkPnz5zNs2DAzmyQiIuLxtP8W8RwK6CJe6quvvmLq1KlVt2+44QYARowYwTPPPMObb77Jq6++yvXXX8/BgwdZvXo1DoeD5ORkRo4cyQsvvMCvfvUrxowZwz/+8Q/S09MBiI2N5aGHHmLt2rWsXbtWO3gREZFmpP23iGdTQBfxUnUdIrdp0yZGjhwJGDv6zz77jK5du3LllVdisVjw9fXlsssuY+/evWRnZ3P55ZcDMH78eAA+/PBDYmNjAejYsSOlpaWt2CIRERHPp/23iGfTKu4iUsOuXbsA+O677+jfvz/9+vWrOjzObrfz/fffExkZSb9+/di5cycA69atY9WqVQBYLBZzChcREfFi2n+LeAaNoIt4qdqHyAGUlpby/vvv8/rrrxMYGMjixYtp3749X3/9NYmJidjtdm677TZiY2P5r//6L+bNm8eyZcsICAjg+eefJy0tzaTWiIiIeAftv0U8m8XlcrnMLkJE3MPUqVOZP38+/fr1M7sUERERaSLtv0U8hw5xFxEREREREXEDGkEXERERERERcQMaQRcRERERERFxAwroIiIiIiIiIm5AAV1ERERERETEDSigi4iIiIiIiLgBBXQRERERERERN/D/AUcIgW4XeFP5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(ncols=2, figsize=(14, 4))\n",
    "df1 = (history[['accuracy', 'val_accuracy']]\n",
    "       .rename(columns={'accuracy': 'Training',\n",
    "                        'val_accuracy': 'Validation'}))\n",
    "df1.plot(ax=axes[0], title='Accuracy', xlim=(1, len(history)))\n",
    "\n",
    "axes[0].axvline(df.val_accuracy.idxmax(), ls='--', lw=1, c='k')\n",
    "\n",
    "df2 = (history[['AUC', 'val_AUC']]\n",
    "       .rename(columns={'AUC': 'Training',\n",
    "                        'val_AUC': 'Validation'}))\n",
    "df2.plot(ax=axes[1], title='Area under the ROC Curve', xlim=(1, len(history)))\n",
    "\n",
    "axes[1].axvline(df.val_AUC.idxmax(), ls='--', lw=1, c='k')\n",
    "\n",
    "for i in [0, 1]:\n",
    "    axes[i].set_xlabel('Epoch')\n",
    "\n",
    "sns.despine()\n",
    "fig.tight_layout()\n",
    "fig.savefig(results_path / 'rnn_imdb_cv', dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T01:37:18.344545Z",
     "start_time": "2020-06-22T01:37:04.465577Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 1)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_score = rnn.predict(X_test_padded)\n",
    "y_score.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-22T01:37:18.362454Z",
     "start_time": "2020-06-22T01:37:18.347568Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9410918336"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_score=y_score.squeeze(), y_true=y_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:ml4t-dl]",
   "language": "python",
   "name": "conda-env-ml4t-dl-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
